lsst.ctrl.bps.drivers INFO: DISCLAIMER: All values regarding memory consumption reported below are approximate and may not accurately reflect actual memory usage by the bps process. lsst.ctrl.bps.drivers INFO: Starting submission process lsst.ctrl.bps.drivers INFO: Initializing execution environment lsst.ctrl.bps.drivers INFO: Initializing execution environment completed: Took 3.2673 seconds; current memory usage: 0.181 Gibyte, delta: 0.042 Gibyte, peak delta: 0.054 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.192 Gibyte (main), 0.000 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting acquire stage (generating and/or reading quantum graph) lsst.ctrl.bps.pre_transform INFO: Copying quantum graph from 'pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed copying quantum graph: Took 0.0082 seconds lsst.ctrl.bps.pre_transform INFO: Backing up quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed backing up quantum graph: Took 0.0247 seconds lsst.ctrl.bps.pre_transform INFO: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_mpexec/g1ce94f1343+c79f27626b/bin/pipetask --long-log --log-level=VERBOSE update-graph-run /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/pipeline_orig.qgraph DEEP/20220823/B1d/science#step1/20240307T072452Z /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/pipeline.qgraph lsst.ctrl.bps.pre_transform INFO: Reading quantum graph from '/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/pipeline.qgraph' lsst.ctrl.bps.pre_transform INFO: Completed reading quantum graph: Took 5.6658 seconds lsst.ctrl.bps.drivers INFO: Acquire stage completed: Took 15.5629 seconds; current memory usage: 0.377 Gibyte, delta: 0.196 Gibyte, peak delta: 0.185 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.377 Gibyte (main), 0.362 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting cluster stage (grouping quanta into jobs) lsst.ctrl.bps.drivers INFO: Cluster stage completed: Took 0.0238 seconds; current memory usage: 0.377 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.377 Gibyte (main), 0.362 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: ClusteredQuantumGraph contains 930 cluster(s) lsst.ctrl.bps.drivers INFO: Starting transform stage (creating generic workflow) lsst.ctrl.bps.drivers INFO: Generic workflow name 'DEEP_20220823_B1d_science#step1_20240307T072452Z' lsst.ctrl.bps.drivers INFO: Transform stage completed: Took 0.1856 seconds; current memory usage: 0.377 Gibyte, delta: 0.000 Gibyte, peak delta: 0.000 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.377 Gibyte (main), 0.362 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: GenericWorkflow contains 932 job(s) (including final) lsst.ctrl.bps.drivers INFO: Starting prepare stage (creating specific implementation of workflow) parsl.addresses ERROR: Ignoring failure to fetch address from interface eno2 Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 111, in get_all_addresses s_addresses.add(address_by_interface(interface)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py", line 1033, in wrapper retval = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/addresses.py", line 93, in address_by_interface return socket.inet_ntoa(fcntl.ioctl( ^^^^^^^^^^^^ OSError: [Errno 99] Cannot assign requested address lsst.ctrl.bps.parsl INFO: Writing workflow with ID=/mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z lsst.ctrl.bps.drivers INFO: Prepare stage completed: Took 0.3079 seconds; current memory usage: 0.379 Gibyte, delta: 0.002 Gibyte, peak delta: 0.002 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.379 Gibyte (main), 0.362 Gibyte (largest child process) lsst.ctrl.bps.drivers INFO: Starting submit stage lsst.ctrl.bps.submit INFO: Submitting run to a workflow management system for execution parsl.dataflow.rundirs DEBUG: Parsl run initializing in rundir: runinfo/000 parsl.dataflow.dflow INFO: Starting DataFlowKernel with config Config( app_cache=True, checkpoint_files=None, checkpoint_mode='task_exit', checkpoint_period=None, executors=[MultiHighThroughputExecutor()], garbage_collect=True, initialize_logging=True, internal_tasks_max_threads=10, max_idletime=120.0, monitoring=None, retries=1, retry_handler=None, run_dir='runinfo', strategy='simple', usage_tracking=False ) parsl.dataflow.dflow INFO: Parsl version: 2023.06.12 parsl.usage_tracking.usage DEBUG: Tracking status: False parsl.dataflow.dflow INFO: Run id is: 5789910a-28d4-47d7-8ee2-2b438756e7e2 parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/typeguard/__init__.py parsl.dataflow.dflow DEBUG: Considering candidate for workflow name: /mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/share/eups/Linux64/ctrl_bps_parsl/g145af14111+5b908e21bc/python/lsst/ctrl/bps/parsl/workflow.py parsl.dataflow.dflow DEBUG: Using workflow.py as workflow name parsl.dataflow.memoization INFO: App caching initialized parsl.dataflow.strategy DEBUG: Scaling strategy: simple parsl.executors.high_throughput.executor DEBUG: Starting queue management thread parsl.executors.high_throughput.executor DEBUG: queue management worker starting parsl.executors.high_throughput.executor DEBUG: Started queue management thread Submit dir: /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z parsl.executors.high_throughput.executor DEBUG: Created management thread: parsl.executors.high_throughput.executor DEBUG: Launch command: process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id={block_id} --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn parsl.executors.high_throughput.executor DEBUG: Starting HighThroughputExecutor with provider: parsl.executors.status_handling INFO: Scaling out by 1 blocks parsl.executors.status_handling INFO: Allocated block ID 0 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-0 2024-03-07 07:25:39 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-0 2024-03-07 07:25:39 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:39 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 07:25:39 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:39 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=0 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 to local parsl.providers.local.local DEBUG: Launching in remote mode 2024-03-07 07:25:39 proc_lsst.multi:170 [INFO] [multi] job_id 21546 proc_lsst.multi INFO: [multi] job_id 21546 2024-03-07 07:25:39 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:39 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:39 proc_lsst.multi:178 [INFO] [multi] provider local accepted submit and returned 21546 proc_lsst.multi INFO: [multi] provider local accepted submit and returned 21546 parsl.executors.status_handling DEBUG: Launched block 0 on executor multi with job ID 21546 parsl.dataflow.job_status_poller DEBUG: Adding executor multi parsl.dataflow.dflow DEBUG: Task 0 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 0 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 0 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 0 has memoization hash bc1feaeac694e25a0ff8a10901da94b4 parsl.dataflow.memoization INFO: Task 0 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad48a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 0 try 0 launched on executor multi with executor id 1 parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/be798d22-6cbd-4fcc-a8e3-525cf8e9396e_isr_1124831_45.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/be798d22-6cbd-4fcc-a8e3-525cf8e9396e_isr_1124831_45.stderr parsl.dataflow.dflow DEBUG: Task 1 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 1 submitted for App characterizeImage, waiting on task 0 parsl.dataflow.dflow DEBUG: Task 1 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 1 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 2 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 2 submitted for App calibrate, waiting on task 1 parsl.dataflow.dflow DEBUG: Task 2 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 2 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 3 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 3 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 3 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 3 has memoization hash 7da6b142842a48fd3ab56a904b7871d6 parsl.dataflow.memoization INFO: Task 3 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad49300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 3 try 0 launched on executor multi with executor id 2 parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/fed63d7b-eb74-4cd5-9884-d90b39b6c920_isr_1124831_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/fed63d7b-eb74-4cd5-9884-d90b39b6c920_isr_1124831_3.stderr parsl.dataflow.dflow DEBUG: Task 4 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 4 submitted for App characterizeImage, waiting on task 3 parsl.dataflow.dflow DEBUG: Task 4 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 4 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 5 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 5 submitted for App calibrate, waiting on task 4 parsl.dataflow.dflow DEBUG: Task 5 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 5 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 6 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 6 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 6 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 6 has memoization hash ef2d0c2919e8e66f0fb8a9f5a688fdbb parsl.dataflow.memoization INFO: Task 6 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad49580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 6 try 0 launched on executor multi with executor id 3 parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/06ab3d96-5d3e-408a-9f33-6234e8a873bc_isr_1124704_2.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/06ab3d96-5d3e-408a-9f33-6234e8a873bc_isr_1124704_2.stderr parsl.dataflow.dflow DEBUG: Task 7 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 7 submitted for App characterizeImage, waiting on task 6 parsl.dataflow.dflow DEBUG: Task 7 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 7 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 8 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 8 submitted for App calibrate, waiting on task 7 parsl.dataflow.dflow DEBUG: Task 8 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 8 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 9 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 9 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 9 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 9 has memoization hash a62f3dfb77aad1f9e43eb19cfa71cf8f parsl.dataflow.memoization INFO: Task 9 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad49760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 9 try 0 launched on executor multi with executor id 4 parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/49552ea2-fcc4-413e-bcd1-ea8d5630860e_isr_1124831_5.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/49552ea2-fcc4-413e-bcd1-ea8d5630860e_isr_1124831_5.stderr parsl.dataflow.dflow DEBUG: Task 10 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 10 submitted for App characterizeImage, waiting on task 9 parsl.dataflow.dflow DEBUG: Task 10 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 10 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 11 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 11 submitted for App calibrate, waiting on task 10 parsl.dataflow.dflow DEBUG: Task 11 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 11 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 12 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 12 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 12 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 12 has memoization hash d389ad3f8069be8fd5c3ae80c2bbab38 parsl.dataflow.memoization INFO: Task 12 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad49c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 12 try 0 launched on executor multi with executor id 5 parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/e3fa8b93-8332-4852-9682-422ab581354a_isr_1124828_28.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/e3fa8b93-8332-4852-9682-422ab581354a_isr_1124828_28.stderr parsl.dataflow.dflow DEBUG: Task 13 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 13 submitted for App characterizeImage, waiting on task 12 parsl.dataflow.dflow DEBUG: Task 13 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 13 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 14 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 14 submitted for App calibrate, waiting on task 13 parsl.dataflow.dflow DEBUG: Task 14 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 14 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 15 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 15 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 15 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 15 has memoization hash 62dd640c64bd61629b0ebf31870d9ab5 parsl.dataflow.memoization INFO: Task 15 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 15 try 0 launched on executor multi with executor id 6 parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/75b20162-2ace-466f-904b-dd59f917d480_isr_1124828_2.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/75b20162-2ace-466f-904b-dd59f917d480_isr_1124828_2.stderr parsl.dataflow.dflow DEBUG: Task 16 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 16 submitted for App characterizeImage, waiting on task 15 parsl.dataflow.dflow DEBUG: Task 16 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 16 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 17 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 17 submitted for App calibrate, waiting on task 16 parsl.dataflow.dflow DEBUG: Task 17 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 17 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 18 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 18 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 18 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 18 has memoization hash 5e121582dfd2903ece176aeff4354383 parsl.dataflow.memoization INFO: Task 18 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 18 try 0 launched on executor multi with executor id 7 parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f8fd87f4-b505-474b-81da-78b7f150b582_isr_1124710_37.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f8fd87f4-b505-474b-81da-78b7f150b582_isr_1124710_37.stderr parsl.dataflow.dflow DEBUG: Task 19 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 19 submitted for App characterizeImage, waiting on task 18 parsl.dataflow.dflow DEBUG: Task 19 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 19 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 20 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 20 submitted for App calibrate, waiting on task 19 parsl.dataflow.dflow DEBUG: Task 20 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 20 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 21 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 21 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 21 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 21 has memoization hash 794f7f9ed9b6535800c9dd298e5849c0 parsl.dataflow.memoization INFO: Task 21 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 21 try 0 launched on executor multi with executor id 8 parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/9707668c-cbdf-474f-b1f1-cb4ff2bd057b_isr_1124828_21.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/9707668c-cbdf-474f-b1f1-cb4ff2bd057b_isr_1124828_21.stderr parsl.dataflow.dflow DEBUG: Task 22 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 22 submitted for App characterizeImage, waiting on task 21 parsl.dataflow.dflow DEBUG: Task 22 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 22 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 23 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 23 submitted for App calibrate, waiting on task 22 parsl.dataflow.dflow DEBUG: Task 23 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 23 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 24 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 24 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 24 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 24 has memoization hash f653dc433b00e6432de5582b5b78d672 parsl.dataflow.memoization INFO: Task 24 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 24 try 0 launched on executor multi with executor id 9 parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/190f9bdf-b788-48a8-aefa-a67390fd62d8_isr_1124828_51.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/190f9bdf-b788-48a8-aefa-a67390fd62d8_isr_1124828_51.stderr parsl.dataflow.dflow DEBUG: Task 25 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 25 submitted for App characterizeImage, waiting on task 24 parsl.dataflow.dflow DEBUG: Task 25 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 25 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 26 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 26 submitted for App calibrate, waiting on task 25 parsl.dataflow.dflow DEBUG: Task 26 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 26 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 27 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 27 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 27 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 27 has memoization hash 63322d5bc26b466ad06ccbf967313bd3 parsl.dataflow.memoization INFO: Task 27 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 27 try 0 launched on executor multi with executor id 10 parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/71c8e0b6-fdb6-4f95-a1a8-4fd17f06335f_isr_1124707_49.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/71c8e0b6-fdb6-4f95-a1a8-4fd17f06335f_isr_1124707_49.stderr parsl.dataflow.dflow DEBUG: Task 28 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 28 submitted for App characterizeImage, waiting on task 27 parsl.dataflow.dflow DEBUG: Task 28 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 28 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 29 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 29 submitted for App calibrate, waiting on task 28 parsl.dataflow.dflow DEBUG: Task 29 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 29 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 30 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 30 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 30 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 30 has memoization hash ab66cc26045422a6c9aa1b1fca2b87c1 parsl.dataflow.memoization INFO: Task 30 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 30 try 0 launched on executor multi with executor id 11 parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/09762abf-0aeb-4d01-ae6e-441eb698f01c_isr_1124710_16.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/09762abf-0aeb-4d01-ae6e-441eb698f01c_isr_1124710_16.stderr parsl.dataflow.dflow DEBUG: Task 31 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 31 submitted for App characterizeImage, waiting on task 30 parsl.dataflow.dflow DEBUG: Task 31 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 31 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 32 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 32 submitted for App calibrate, waiting on task 31 parsl.dataflow.dflow DEBUG: Task 32 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 32 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 33 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 33 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 33 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 33 has memoization hash 297e8819f84fe5b72d8e6817ce58c74b parsl.dataflow.memoization INFO: Task 33 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 33 try 0 launched on executor multi with executor id 12 parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/e1230913-bf8f-4100-a371-2b1a0948bfef_isr_1124710_25.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/e1230913-bf8f-4100-a371-2b1a0948bfef_isr_1124710_25.stderr parsl.dataflow.dflow DEBUG: Task 34 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 34 submitted for App characterizeImage, waiting on task 33 parsl.dataflow.dflow DEBUG: Task 34 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 34 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 35 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 35 submitted for App calibrate, waiting on task 34 parsl.dataflow.dflow DEBUG: Task 35 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 35 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 36 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 36 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 36 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 36 has memoization hash b4f645c0a85e302577f3dbc3b07bdced parsl.dataflow.memoization INFO: Task 36 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 36 try 0 launched on executor multi with executor id 13 parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0bf287b9-7b36-4fe0-bea4-fde91691012f_isr_1124707_25.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0bf287b9-7b36-4fe0-bea4-fde91691012f_isr_1124707_25.stderr parsl.dataflow.dflow DEBUG: Task 37 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 37 submitted for App characterizeImage, waiting on task 36 parsl.dataflow.dflow DEBUG: Task 37 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 37 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 38 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 38 submitted for App calibrate, waiting on task 37 parsl.dataflow.dflow DEBUG: Task 38 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 38 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 39 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 39 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 39 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 39 has memoization hash 5ae00c1aa3027691bdf98dd30b5a9ccb parsl.dataflow.memoization INFO: Task 39 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 39 try 0 launched on executor multi with executor id 14 parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/9fe215b5-501b-4963-8de5-ebc04249b4fe_isr_1124828_32.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/9fe215b5-501b-4963-8de5-ebc04249b4fe_isr_1124828_32.stderr parsl.dataflow.dflow DEBUG: Task 40 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 40 submitted for App characterizeImage, waiting on task 39 parsl.dataflow.dflow DEBUG: Task 40 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 40 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 41 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 41 submitted for App calibrate, waiting on task 40 parsl.dataflow.dflow DEBUG: Task 41 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 41 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 42 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 42 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 42 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 42 has memoization hash 58b50830b2b8da797f17f9f658440454 parsl.dataflow.memoization INFO: Task 42 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 42 try 0 launched on executor multi with executor id 15 parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/40ceabf1-a82e-4c7f-af40-68837de7c439_isr_1124707_20.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/40ceabf1-a82e-4c7f-af40-68837de7c439_isr_1124707_20.stderr parsl.dataflow.dflow DEBUG: Task 43 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 43 submitted for App characterizeImage, waiting on task 42 parsl.dataflow.dflow DEBUG: Task 43 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 43 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 44 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 44 submitted for App calibrate, waiting on task 43 parsl.dataflow.dflow DEBUG: Task 44 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 44 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 45 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 45 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 45 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 45 has memoization hash a89c09619461a3bdbde80380e0459f88 parsl.dataflow.memoization INFO: Task 45 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad640e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 45 try 0 launched on executor multi with executor id 16 parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/bd1198d9-6de0-4c50-b46e-3fc790653593_isr_1124831_33.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/bd1198d9-6de0-4c50-b46e-3fc790653593_isr_1124831_33.stderr parsl.dataflow.dflow DEBUG: Task 46 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 46 submitted for App characterizeImage, waiting on task 45 parsl.dataflow.dflow DEBUG: Task 46 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 46 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 47 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 47 submitted for App calibrate, waiting on task 46 parsl.dataflow.dflow DEBUG: Task 47 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 47 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 48 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 48 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 48 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 48 has memoization hash 716ce532f28c0e2a357f2c72b9162d44 parsl.dataflow.memoization INFO: Task 48 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 48 try 0 launched on executor multi with executor id 17 parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/7fa5eda9-3f3e-44c7-a9a3-cc94305469f2_isr_1124828_34.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/7fa5eda9-3f3e-44c7-a9a3-cc94305469f2_isr_1124828_34.stderr parsl.dataflow.dflow DEBUG: Task 49 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 49 submitted for App characterizeImage, waiting on task 48 parsl.dataflow.dflow DEBUG: Task 49 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 49 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 50 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 50 submitted for App calibrate, waiting on task 49 parsl.dataflow.dflow DEBUG: Task 50 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 50 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 51 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 51 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 51 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 51 has memoization hash 91f07a27982fc8ee308716f48c2405c1 parsl.dataflow.memoization INFO: Task 51 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 51 try 0 launched on executor multi with executor id 18 parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/169d9003-00e9-4f1d-b1fa-ca1f17698f63_isr_1124707_41.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/169d9003-00e9-4f1d-b1fa-ca1f17698f63_isr_1124707_41.stderr parsl.dataflow.dflow DEBUG: Task 52 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 52 submitted for App characterizeImage, waiting on task 51 parsl.dataflow.dflow DEBUG: Task 52 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 52 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 53 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 53 submitted for App calibrate, waiting on task 52 parsl.dataflow.dflow DEBUG: Task 53 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 53 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 54 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 54 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 54 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 54 has memoization hash 017cc84768d23a99ae319417377df602 parsl.dataflow.memoization INFO: Task 54 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 54 try 0 launched on executor multi with executor id 19 parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/da8fc843-509a-47a0-b875-716810c3931f_isr_1124831_6.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/da8fc843-509a-47a0-b875-716810c3931f_isr_1124831_6.stderr parsl.dataflow.dflow DEBUG: Task 55 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 55 submitted for App characterizeImage, waiting on task 54 parsl.dataflow.dflow DEBUG: Task 55 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 55 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 56 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 56 submitted for App calibrate, waiting on task 55 parsl.dataflow.dflow DEBUG: Task 56 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 56 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 57 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 57 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 57 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 57 has memoization hash e5ee78ef563a024255feed1348500e43 parsl.dataflow.memoization INFO: Task 57 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 57 try 0 launched on executor multi with executor id 20 parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/10c5a42a-1db1-431f-b032-acfa529b0f5b_isr_1124710_30.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/10c5a42a-1db1-431f-b032-acfa529b0f5b_isr_1124710_30.stderr parsl.dataflow.dflow DEBUG: Task 58 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 58 submitted for App characterizeImage, waiting on task 57 parsl.dataflow.dflow DEBUG: Task 58 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 58 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 59 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 59 submitted for App calibrate, waiting on task 58 parsl.dataflow.dflow DEBUG: Task 59 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 59 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 60 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 60 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 60 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 60 has memoization hash 725c7840f1aca20450ecda274a42946c parsl.dataflow.memoization INFO: Task 60 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad65080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 60 try 0 launched on executor multi with executor id 21 parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f1d2229e-8018-4fb0-abfa-f6f7c71c0a9e_isr_1124831_40.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f1d2229e-8018-4fb0-abfa-f6f7c71c0a9e_isr_1124831_40.stderr parsl.dataflow.dflow DEBUG: Task 61 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 61 submitted for App characterizeImage, waiting on task 60 parsl.dataflow.dflow DEBUG: Task 61 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 61 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 62 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 62 submitted for App calibrate, waiting on task 61 parsl.dataflow.dflow DEBUG: Task 62 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 62 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 63 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 63 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 63 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 63 has memoization hash 5e5fc2df59618c2cc66020ed20703b1f parsl.dataflow.memoization INFO: Task 63 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad653a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 63 try 0 launched on executor multi with executor id 22 parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6325b6a6-20e9-46dd-8607-76dec1ab895e_isr_1124704_9.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6325b6a6-20e9-46dd-8607-76dec1ab895e_isr_1124704_9.stderr parsl.dataflow.dflow DEBUG: Task 64 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 64 submitted for App characterizeImage, waiting on task 63 parsl.dataflow.dflow DEBUG: Task 64 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 64 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 65 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 65 submitted for App calibrate, waiting on task 64 parsl.dataflow.dflow DEBUG: Task 65 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 65 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 66 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 66 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 66 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 66 has memoization hash c7fc5a91626fcd2fb3effe45fa979b35 parsl.dataflow.memoization INFO: Task 66 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad656c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 66 try 0 launched on executor multi with executor id 23 parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/d3f5f44b-8a94-455d-bf37-f6bc6537ed41_isr_1124828_59.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/d3f5f44b-8a94-455d-bf37-f6bc6537ed41_isr_1124828_59.stderr parsl.dataflow.dflow DEBUG: Task 67 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 67 submitted for App characterizeImage, waiting on task 66 parsl.dataflow.dflow DEBUG: Task 67 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 67 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 68 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 68 submitted for App calibrate, waiting on task 67 parsl.dataflow.dflow DEBUG: Task 68 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 68 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 69 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 69 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 69 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 69 has memoization hash b900a93eabec86277bbb09fa365f4155 parsl.dataflow.memoization INFO: Task 69 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad659e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 69 try 0 launched on executor multi with executor id 24 parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/5d53417c-92cb-4b45-a959-9353fe490628_isr_1124831_47.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/5d53417c-92cb-4b45-a959-9353fe490628_isr_1124831_47.stderr parsl.dataflow.dflow DEBUG: Task 70 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 70 submitted for App characterizeImage, waiting on task 69 parsl.dataflow.dflow DEBUG: Task 70 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 70 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 71 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 71 submitted for App calibrate, waiting on task 70 parsl.dataflow.dflow DEBUG: Task 71 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 71 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 72 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 72 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 72 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 72 has memoization hash cc9107800275a5d92fda10ac72f8b392 parsl.dataflow.memoization INFO: Task 72 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad65d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 72 try 0 launched on executor multi with executor id 25 parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/aa1a792e-8052-4151-9b4e-ece23ef0cdb9_isr_1124831_58.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/aa1a792e-8052-4151-9b4e-ece23ef0cdb9_isr_1124831_58.stderr parsl.dataflow.dflow DEBUG: Task 73 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 73 submitted for App characterizeImage, waiting on task 72 parsl.dataflow.dflow DEBUG: Task 73 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 73 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 74 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 74 submitted for App calibrate, waiting on task 73 parsl.dataflow.dflow DEBUG: Task 74 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 74 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 75 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 75 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 75 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 75 has memoization hash 42bddae0a2a3e8032bf5a999a25cac32 parsl.dataflow.memoization INFO: Task 75 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad66020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 75 try 0 launched on executor multi with executor id 26 parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/5e6a4de0-dc4f-4820-808a-b8bd6c853f55_isr_1124831_30.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/5e6a4de0-dc4f-4820-808a-b8bd6c853f55_isr_1124831_30.stderr parsl.dataflow.dflow DEBUG: Task 76 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 76 submitted for App characterizeImage, waiting on task 75 parsl.dataflow.dflow DEBUG: Task 76 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 76 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 77 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 77 submitted for App calibrate, waiting on task 76 parsl.dataflow.dflow DEBUG: Task 77 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 77 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 78 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 78 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 78 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 78 has memoization hash e6d94e9fa5c4d96d7f4588bef804f2a8 parsl.dataflow.memoization INFO: Task 78 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad66340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 78 try 0 launched on executor multi with executor id 27 parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/610a69d0-70fe-441d-8543-5dc81d726783_isr_1124710_18.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/610a69d0-70fe-441d-8543-5dc81d726783_isr_1124710_18.stderr parsl.dataflow.dflow DEBUG: Task 79 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 79 submitted for App characterizeImage, waiting on task 78 parsl.dataflow.dflow DEBUG: Task 79 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 79 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 80 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 80 submitted for App calibrate, waiting on task 79 parsl.dataflow.dflow DEBUG: Task 80 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 80 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 81 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 81 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 81 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 81 has memoization hash dec2b2ed52bdb96a3a342b853067ccc8 parsl.dataflow.memoization INFO: Task 81 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad66660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 81 try 0 launched on executor multi with executor id 28 parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/21020276-238d-45ab-9a6a-785b5f99ab25_isr_1124828_61.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/21020276-238d-45ab-9a6a-785b5f99ab25_isr_1124828_61.stderr parsl.dataflow.dflow DEBUG: Task 82 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 82 submitted for App characterizeImage, waiting on task 81 parsl.dataflow.dflow DEBUG: Task 82 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 82 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 83 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 83 submitted for App calibrate, waiting on task 82 parsl.dataflow.dflow DEBUG: Task 83 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 83 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 84 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 84 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 84 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 84 has memoization hash 479bdbae7bbbf235f0e2f54432c07bff parsl.dataflow.memoization INFO: Task 84 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad66980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 84 try 0 launched on executor multi with executor id 29 parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/599ab672-1375-4e4a-bee9-ab2e9e5a3621_isr_1124710_55.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/599ab672-1375-4e4a-bee9-ab2e9e5a3621_isr_1124710_55.stderr parsl.dataflow.dflow DEBUG: Task 85 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 85 submitted for App characterizeImage, waiting on task 84 parsl.dataflow.dflow DEBUG: Task 85 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 85 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 86 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 86 submitted for App calibrate, waiting on task 85 parsl.dataflow.dflow DEBUG: Task 86 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 86 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 87 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 87 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 87 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 87 has memoization hash 8a875715f8a3e04c976df800712abf73 parsl.dataflow.memoization INFO: Task 87 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad66ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 87 try 0 launched on executor multi with executor id 30 parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a3500b1b-0adb-4295-a23c-fdaf75aeabbe_isr_1124710_41.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a3500b1b-0adb-4295-a23c-fdaf75aeabbe_isr_1124710_41.stderr parsl.dataflow.dflow DEBUG: Task 88 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 88 submitted for App characterizeImage, waiting on task 87 parsl.dataflow.dflow DEBUG: Task 88 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 88 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 89 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 89 submitted for App calibrate, waiting on task 88 parsl.dataflow.dflow DEBUG: Task 89 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 89 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 90 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 90 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 90 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 90 has memoization hash 1c5fd6a854b03976a3ad7d39e5f71483 parsl.dataflow.memoization INFO: Task 90 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad66fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 90 try 0 launched on executor multi with executor id 31 parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/974d1369-ab1a-42ae-ab64-4ec601303b03_isr_1124831_25.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/974d1369-ab1a-42ae-ab64-4ec601303b03_isr_1124831_25.stderr parsl.dataflow.dflow DEBUG: Task 91 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 91 submitted for App characterizeImage, waiting on task 90 parsl.dataflow.dflow DEBUG: Task 91 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 91 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 92 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 92 submitted for App calibrate, waiting on task 91 parsl.dataflow.dflow DEBUG: Task 92 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 92 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 93 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 93 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 93 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 93 has memoization hash a3cde5038abff51790b89d7780af9088 parsl.dataflow.memoization INFO: Task 93 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad672e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 93 try 0 launched on executor multi with executor id 32 parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/86357f3e-8744-402b-a680-b70ca28a3723_isr_1124831_23.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/86357f3e-8744-402b-a680-b70ca28a3723_isr_1124831_23.stderr parsl.dataflow.dflow DEBUG: Task 94 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 94 submitted for App characterizeImage, waiting on task 93 parsl.dataflow.dflow DEBUG: Task 94 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 94 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 95 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 95 submitted for App calibrate, waiting on task 94 parsl.dataflow.dflow DEBUG: Task 95 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 95 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 96 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 96 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 96 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 96 has memoization hash 35cc37e46b5bcbee280af13913b54e31 parsl.dataflow.memoization INFO: Task 96 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 96 try 0 launched on executor multi with executor id 33 parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a3f8f162-f2d8-4565-b902-bbc12da8754e_isr_1124704_14.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a3f8f162-f2d8-4565-b902-bbc12da8754e_isr_1124704_14.stderr parsl.dataflow.dflow DEBUG: Task 97 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 97 submitted for App characterizeImage, waiting on task 96 parsl.dataflow.dflow DEBUG: Task 97 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 97 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 98 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 98 submitted for App calibrate, waiting on task 97 parsl.dataflow.dflow DEBUG: Task 98 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 98 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 99 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 99 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 99 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 99 has memoization hash 3c766e3d4efcadb4e42a50a8959451f4 parsl.dataflow.memoization INFO: Task 99 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 99 try 0 launched on executor multi with executor id 34 parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/ce9f6d31-a10a-4d39-a64f-3690be8b96aa_isr_1124831_27.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/ce9f6d31-a10a-4d39-a64f-3690be8b96aa_isr_1124831_27.stderr parsl.dataflow.dflow DEBUG: Task 100 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 100 submitted for App characterizeImage, waiting on task 99 parsl.dataflow.dflow DEBUG: Task 100 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 100 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 101 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 101 submitted for App calibrate, waiting on task 100 parsl.dataflow.dflow DEBUG: Task 101 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 101 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 102 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 102 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 102 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 102 has memoization hash f00caba3b5f69de4e71937a3a97f79a0 parsl.dataflow.memoization INFO: Task 102 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67c40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 102 try 0 launched on executor multi with executor id 35 parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/ff449df6-c8bb-455f-a9a3-e280d77c856f_isr_1124710_22.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/ff449df6-c8bb-455f-a9a3-e280d77c856f_isr_1124710_22.stderr parsl.dataflow.dflow DEBUG: Task 103 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 103 submitted for App characterizeImage, waiting on task 102 parsl.dataflow.dflow DEBUG: Task 103 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 103 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 104 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 104 submitted for App calibrate, waiting on task 103 parsl.dataflow.dflow DEBUG: Task 104 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 104 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 105 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 105 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 105 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 105 has memoization hash 70eda686be7278b839003d320fa33ad3 parsl.dataflow.memoization INFO: Task 105 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 105 try 0 launched on executor multi with executor id 36 parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/97ad3848-dc65-46f8-a45a-e8b1630e68cb_isr_1124710_31.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/97ad3848-dc65-46f8-a45a-e8b1630e68cb_isr_1124710_31.stderr parsl.dataflow.dflow DEBUG: Task 106 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 106 submitted for App characterizeImage, waiting on task 105 parsl.dataflow.dflow DEBUG: Task 106 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 106 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 107 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 107 submitted for App calibrate, waiting on task 106 parsl.dataflow.dflow DEBUG: Task 107 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 107 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 108 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 108 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 108 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 108 has memoization hash 33e5aaf7429ab1b47db22a978d900a0e parsl.dataflow.memoization INFO: Task 108 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 108 try 0 launched on executor multi with executor id 37 parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7528194f-fcfe-4070-bc9a-1ddfd23e7d14_isr_1124831_42.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7528194f-fcfe-4070-bc9a-1ddfd23e7d14_isr_1124831_42.stderr parsl.dataflow.dflow DEBUG: Task 109 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 109 submitted for App characterizeImage, waiting on task 108 parsl.dataflow.dflow DEBUG: Task 109 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 109 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 110 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 110 submitted for App calibrate, waiting on task 109 parsl.dataflow.dflow DEBUG: Task 110 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 110 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 111 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 111 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 111 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 111 has memoization hash a3b0d131e22a6ef73128259709ab16a7 parsl.dataflow.memoization INFO: Task 111 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 111 try 0 launched on executor multi with executor id 38 parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ce60ec52-85df-4635-8a3b-e7723a09ced9_isr_1124704_49.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ce60ec52-85df-4635-8a3b-e7723a09ced9_isr_1124704_49.stderr parsl.dataflow.dflow DEBUG: Task 112 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 112 submitted for App characterizeImage, waiting on task 111 parsl.dataflow.dflow DEBUG: Task 112 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 112 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 113 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 113 submitted for App calibrate, waiting on task 112 parsl.dataflow.dflow DEBUG: Task 113 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 113 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 114 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 114 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 114 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 114 has memoization hash 04ce89e007fb87d1c76d22d94d8cac0f parsl.dataflow.memoization INFO: Task 114 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 114 try 0 launched on executor multi with executor id 39 parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/dddd2040-87c4-4663-81ff-6f96f59dcd69_isr_1124828_58.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/dddd2040-87c4-4663-81ff-6f96f59dcd69_isr_1124828_58.stderr parsl.dataflow.dflow DEBUG: Task 115 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 115 submitted for App characterizeImage, waiting on task 114 parsl.dataflow.dflow DEBUG: Task 115 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 115 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 116 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 116 submitted for App calibrate, waiting on task 115 parsl.dataflow.dflow DEBUG: Task 116 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 116 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 117 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 117 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 117 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 117 has memoization hash 186d62b5ef3c41148b97421c317f942f parsl.dataflow.memoization INFO: Task 117 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 117 try 0 launched on executor multi with executor id 40 parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/e15dffb1-ed6f-45b9-ab94-369c2a1d32b9_isr_1124710_9.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/e15dffb1-ed6f-45b9-ab94-369c2a1d32b9_isr_1124710_9.stderr parsl.dataflow.dflow DEBUG: Task 118 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 118 submitted for App characterizeImage, waiting on task 117 parsl.dataflow.dflow DEBUG: Task 118 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 118 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 119 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 119 submitted for App calibrate, waiting on task 118 parsl.dataflow.dflow DEBUG: Task 119 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 119 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 120 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 120 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 120 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 120 has memoization hash 6830f77b060fae2f5e1d6c11e5b6d049 parsl.dataflow.memoization INFO: Task 120 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8cf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 120 try 0 launched on executor multi with executor id 41 parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/80233f77-0ec0-47ea-ba91-481b59196155_isr_1124704_50.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/80233f77-0ec0-47ea-ba91-481b59196155_isr_1124704_50.stderr parsl.dataflow.dflow DEBUG: Task 121 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 121 submitted for App characterizeImage, waiting on task 120 parsl.dataflow.dflow DEBUG: Task 121 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 121 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 122 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 122 submitted for App calibrate, waiting on task 121 parsl.dataflow.dflow DEBUG: Task 122 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 122 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 123 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 123 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 123 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 123 has memoization hash 65806d8abebae3bf2ab0e7977a75dd7f parsl.dataflow.memoization INFO: Task 123 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 123 try 0 launched on executor multi with executor id 42 parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/c145c9c4-7028-4407-9354-43f2322a0060_isr_1124831_28.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/c145c9c4-7028-4407-9354-43f2322a0060_isr_1124831_28.stderr parsl.dataflow.dflow DEBUG: Task 124 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 124 submitted for App characterizeImage, waiting on task 123 parsl.dataflow.dflow DEBUG: Task 124 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 124 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 125 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 125 submitted for App calibrate, waiting on task 124 parsl.dataflow.dflow DEBUG: Task 125 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 125 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 126 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 126 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 126 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 126 has memoization hash 5bf71ad492aab2235c0a6e6848ba93a3 parsl.dataflow.memoization INFO: Task 126 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 126 try 0 launched on executor multi with executor id 43 parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/6b1717a8-ae3c-48ea-ab68-6f4eb357215f_isr_1124828_13.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/6b1717a8-ae3c-48ea-ab68-6f4eb357215f_isr_1124828_13.stderr parsl.dataflow.dflow DEBUG: Task 127 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 127 submitted for App characterizeImage, waiting on task 126 parsl.dataflow.dflow DEBUG: Task 127 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 127 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 128 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 128 submitted for App calibrate, waiting on task 127 parsl.dataflow.dflow DEBUG: Task 128 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 128 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 129 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 129 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 129 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 129 has memoization hash 7e29f93569faa2bfbcf669cbc8f135db parsl.dataflow.memoization INFO: Task 129 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 129 try 0 launched on executor multi with executor id 44 parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/23841145-6556-4fec-8575-70d9b7cf6973_isr_1124704_22.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/23841145-6556-4fec-8575-70d9b7cf6973_isr_1124704_22.stderr parsl.dataflow.dflow DEBUG: Task 130 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 130 submitted for App characterizeImage, waiting on task 129 parsl.dataflow.dflow DEBUG: Task 130 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 130 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 131 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 131 submitted for App calibrate, waiting on task 130 parsl.dataflow.dflow DEBUG: Task 131 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 131 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 132 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 132 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 132 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 132 has memoization hash ddc4d180463e50a51da9744acad21405 parsl.dataflow.memoization INFO: Task 132 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 132 try 0 launched on executor multi with executor id 45 parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/50bdfbcd-5af9-407f-829b-9dd8904c9cdf_isr_1124831_20.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/50bdfbcd-5af9-407f-829b-9dd8904c9cdf_isr_1124831_20.stderr parsl.dataflow.dflow DEBUG: Task 133 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 133 submitted for App characterizeImage, waiting on task 132 parsl.dataflow.dflow DEBUG: Task 133 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 133 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 134 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 134 submitted for App calibrate, waiting on task 133 parsl.dataflow.dflow DEBUG: Task 134 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 134 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 135 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 135 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 135 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 135 has memoization hash c53aa116692d7d8f33dbe9bd64eee1aa parsl.dataflow.memoization INFO: Task 135 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 135 try 0 launched on executor multi with executor id 46 parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/fb3cbdf4-29df-4b9a-a908-1867015d43f1_isr_1124707_36.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/fb3cbdf4-29df-4b9a-a908-1867015d43f1_isr_1124707_36.stderr parsl.dataflow.dflow DEBUG: Task 136 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 136 submitted for App characterizeImage, waiting on task 135 parsl.dataflow.dflow DEBUG: Task 136 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 136 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 137 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 137 submitted for App calibrate, waiting on task 136 parsl.dataflow.dflow DEBUG: Task 137 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 137 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 138 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 138 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 138 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 138 has memoization hash f13ca967b4723fa13aab29f78087918a parsl.dataflow.memoization INFO: Task 138 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 138 try 0 launched on executor multi with executor id 47 parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a973179e-6358-4a1e-a681-986f2af57feb_isr_1124710_33.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a973179e-6358-4a1e-a681-986f2af57feb_isr_1124710_33.stderr parsl.dataflow.dflow DEBUG: Task 139 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 139 submitted for App characterizeImage, waiting on task 138 parsl.dataflow.dflow DEBUG: Task 139 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 139 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 140 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 140 submitted for App calibrate, waiting on task 139 parsl.dataflow.dflow DEBUG: Task 140 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 140 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 141 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 141 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 141 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 141 has memoization hash bfd5db7a9d0001ff039792868ca8084d parsl.dataflow.memoization INFO: Task 141 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 141 try 0 launched on executor multi with executor id 48 parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/53855d15-ac2c-4b49-a0a7-d2d8a556f6aa_isr_1124710_32.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/53855d15-ac2c-4b49-a0a7-d2d8a556f6aa_isr_1124710_32.stderr parsl.dataflow.dflow DEBUG: Task 142 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 142 submitted for App characterizeImage, waiting on task 141 parsl.dataflow.dflow DEBUG: Task 142 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 142 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 143 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 143 submitted for App calibrate, waiting on task 142 parsl.dataflow.dflow DEBUG: Task 143 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 143 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 144 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 144 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 144 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 144 has memoization hash 02e5318cf6ee5b3e28f35b7d98cdeb2f parsl.dataflow.memoization INFO: Task 144 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 144 try 0 launched on executor multi with executor id 49 parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1454a56b-9ca4-4f71-be41-c7e769db658b_isr_1124710_23.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1454a56b-9ca4-4f71-be41-c7e769db658b_isr_1124710_23.stderr parsl.dataflow.dflow DEBUG: Task 145 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 145 submitted for App characterizeImage, waiting on task 144 parsl.dataflow.dflow DEBUG: Task 145 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 145 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 146 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 146 submitted for App calibrate, waiting on task 145 parsl.dataflow.dflow DEBUG: Task 146 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 146 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 147 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 147 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 147 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 147 has memoization hash d8347172d25fc55a442244ece6e6527a parsl.dataflow.memoization INFO: Task 147 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 147 try 0 launched on executor multi with executor id 50 parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/058d57b9-1882-4510-b4b1-a5c42c4e309d_isr_1124831_21.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/058d57b9-1882-4510-b4b1-a5c42c4e309d_isr_1124831_21.stderr parsl.dataflow.dflow DEBUG: Task 148 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 148 submitted for App characterizeImage, waiting on task 147 parsl.dataflow.dflow DEBUG: Task 148 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 148 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 149 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 149 submitted for App calibrate, waiting on task 148 parsl.dataflow.dflow DEBUG: Task 149 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 149 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 150 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 150 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 150 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 150 has memoization hash 19a14f5932fd704d18eddf9e00fe4306 parsl.dataflow.memoization INFO: Task 150 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 150 try 0 launched on executor multi with executor id 51 parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bf7b019f-6a22-4402-be34-2ca09248c900_isr_1124828_26.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bf7b019f-6a22-4402-be34-2ca09248c900_isr_1124828_26.stderr parsl.dataflow.dflow DEBUG: Task 151 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 151 submitted for App characterizeImage, waiting on task 150 parsl.dataflow.dflow DEBUG: Task 151 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 151 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 152 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 152 submitted for App calibrate, waiting on task 151 parsl.dataflow.dflow DEBUG: Task 152 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 152 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 153 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 153 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 153 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 153 has memoization hash 1e92a644d904914066ba5fbbc6999f36 parsl.dataflow.memoization INFO: Task 153 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 153 try 0 launched on executor multi with executor id 52 parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f23d3132-a0f2-4c1e-9200-319b290ba01b_isr_1124831_59.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f23d3132-a0f2-4c1e-9200-319b290ba01b_isr_1124831_59.stderr parsl.dataflow.dflow DEBUG: Task 154 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 154 submitted for App characterizeImage, waiting on task 153 parsl.dataflow.dflow DEBUG: Task 154 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 154 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 155 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 155 submitted for App calibrate, waiting on task 154 parsl.dataflow.dflow DEBUG: Task 155 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 155 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 156 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 156 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 156 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 156 has memoization hash 0e863ca4a6c20783acceaece33363935 parsl.dataflow.memoization INFO: Task 156 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 156 try 0 launched on executor multi with executor id 53 parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/04d9813c-dcd9-483a-ae59-a6a46388a777_isr_1124710_56.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/04d9813c-dcd9-483a-ae59-a6a46388a777_isr_1124710_56.stderr parsl.dataflow.dflow DEBUG: Task 157 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 157 submitted for App characterizeImage, waiting on task 156 parsl.dataflow.dflow DEBUG: Task 157 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 157 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 158 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 158 submitted for App calibrate, waiting on task 157 parsl.dataflow.dflow DEBUG: Task 158 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 158 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 159 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 159 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 159 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 159 has memoization hash ad2311b92ba0b1ccd1f27191c3a0506d parsl.dataflow.memoization INFO: Task 159 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 159 try 0 launched on executor multi with executor id 54 parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/3ceed0e6-cbf7-487c-8e47-c08182d5c563_isr_1124828_62.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/3ceed0e6-cbf7-487c-8e47-c08182d5c563_isr_1124828_62.stderr parsl.dataflow.dflow DEBUG: Task 160 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 160 submitted for App characterizeImage, waiting on task 159 parsl.dataflow.dflow DEBUG: Task 160 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 160 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 161 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 161 submitted for App calibrate, waiting on task 160 parsl.dataflow.dflow DEBUG: Task 161 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 161 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 162 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 162 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 162 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 162 has memoization hash 5ad330097e87aa068b06b3a062c04948 parsl.dataflow.memoization INFO: Task 162 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 162 try 0 launched on executor multi with executor id 55 parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/1b0d04f7-6967-422f-a0fe-1c2a4d82b7e0_isr_1124704_37.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/1b0d04f7-6967-422f-a0fe-1c2a4d82b7e0_isr_1124704_37.stderr parsl.dataflow.dflow DEBUG: Task 163 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 163 submitted for App characterizeImage, waiting on task 162 parsl.dataflow.dflow DEBUG: Task 163 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 163 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 164 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 164 submitted for App calibrate, waiting on task 163 parsl.dataflow.dflow DEBUG: Task 164 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 164 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 165 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 165 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 165 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 165 has memoization hash dfa0339ef63701b0a55a9c5320dbd18d parsl.dataflow.memoization INFO: Task 165 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 165 try 0 launched on executor multi with executor id 56 parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/65f386b0-2218-404f-9372-51860ccc3ecd_isr_1124704_6.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/65f386b0-2218-404f-9372-51860ccc3ecd_isr_1124704_6.stderr parsl.dataflow.dflow DEBUG: Task 166 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 166 submitted for App characterizeImage, waiting on task 165 parsl.dataflow.dflow DEBUG: Task 166 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 166 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 167 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 167 submitted for App calibrate, waiting on task 166 parsl.dataflow.dflow DEBUG: Task 167 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 167 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 168 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 168 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 168 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 168 has memoization hash 0fde7a01ff85f009e8b7cfecad454eab parsl.dataflow.memoization INFO: Task 168 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adac180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 168 try 0 launched on executor multi with executor id 57 parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/66367b51-dcf0-40f8-b949-8f46b99d598f_isr_1124704_25.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/66367b51-dcf0-40f8-b949-8f46b99d598f_isr_1124704_25.stderr parsl.dataflow.dflow DEBUG: Task 169 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 169 submitted for App characterizeImage, waiting on task 168 parsl.dataflow.dflow DEBUG: Task 169 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 169 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 170 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 170 submitted for App calibrate, waiting on task 169 parsl.dataflow.dflow DEBUG: Task 170 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 170 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 171 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 171 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 171 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 171 has memoization hash b8319a449297333abd4804e1040a2a76 parsl.dataflow.memoization INFO: Task 171 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adac4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 171 try 0 launched on executor multi with executor id 58 parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0d77cac8-968a-40aa-a849-4402fea7b425_isr_1124707_30.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0d77cac8-968a-40aa-a849-4402fea7b425_isr_1124707_30.stderr parsl.dataflow.dflow DEBUG: Task 172 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 172 submitted for App characterizeImage, waiting on task 171 parsl.dataflow.dflow DEBUG: Task 172 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 172 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 173 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 173 submitted for App calibrate, waiting on task 172 parsl.dataflow.dflow DEBUG: Task 173 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 173 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 174 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 174 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 174 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 174 has memoization hash 8dc242dedd1e7e528b8d6fdacb5a755c parsl.dataflow.memoization INFO: Task 174 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adac7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 174 try 0 launched on executor multi with executor id 59 parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/357f83ce-6317-4bf6-b911-20c3742381d1_isr_1124831_31.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/357f83ce-6317-4bf6-b911-20c3742381d1_isr_1124831_31.stderr parsl.dataflow.dflow DEBUG: Task 175 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 175 submitted for App characterizeImage, waiting on task 174 parsl.dataflow.dflow DEBUG: Task 175 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 175 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 176 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 176 submitted for App calibrate, waiting on task 175 parsl.dataflow.dflow DEBUG: Task 176 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 176 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 177 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 177 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 177 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 177 has memoization hash ac6bdf82bf5c88423a16f3f01ae5c8f4 parsl.dataflow.memoization INFO: Task 177 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adacae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 177 try 0 launched on executor multi with executor id 60 parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/72c9285a-ed6e-4816-be1d-02757b8e68c2_isr_1124704_60.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/72c9285a-ed6e-4816-be1d-02757b8e68c2_isr_1124704_60.stderr parsl.dataflow.dflow DEBUG: Task 178 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 178 submitted for App characterizeImage, waiting on task 177 parsl.dataflow.dflow DEBUG: Task 178 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 178 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 179 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 179 submitted for App calibrate, waiting on task 178 parsl.dataflow.dflow DEBUG: Task 179 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 179 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 180 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 180 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 180 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 180 has memoization hash eed074dbfc5e9ae679a0a75a918e0628 parsl.dataflow.memoization INFO: Task 180 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adace00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 180 try 0 launched on executor multi with executor id 61 parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/12a317b3-6154-422c-b0c0-8080c4f25834_isr_1124704_58.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/12a317b3-6154-422c-b0c0-8080c4f25834_isr_1124704_58.stderr parsl.dataflow.dflow DEBUG: Task 181 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 181 submitted for App characterizeImage, waiting on task 180 parsl.dataflow.dflow DEBUG: Task 181 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 181 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 182 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 182 submitted for App calibrate, waiting on task 181 parsl.dataflow.dflow DEBUG: Task 182 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 182 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 183 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 183 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 183 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 183 has memoization hash cfa178d55e9579968e46822f2823881d parsl.dataflow.memoization INFO: Task 183 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adad120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 183 try 0 launched on executor multi with executor id 62 parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/b660c004-66d2-456d-926c-20b543acce08_isr_1124828_41.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/b660c004-66d2-456d-926c-20b543acce08_isr_1124828_41.stderr parsl.dataflow.dflow DEBUG: Task 184 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 184 submitted for App characterizeImage, waiting on task 183 parsl.dataflow.dflow DEBUG: Task 184 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 184 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 185 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 185 submitted for App calibrate, waiting on task 184 parsl.dataflow.dflow DEBUG: Task 185 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 185 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 186 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 186 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 186 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 186 has memoization hash 85e69c4ca71fb48d2378601de322ba4c parsl.dataflow.memoization INFO: Task 186 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adad440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 186 try 0 launched on executor multi with executor id 63 parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/b583b7ec-4780-46a6-b61d-1224d3cfd441_isr_1124828_42.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/b583b7ec-4780-46a6-b61d-1224d3cfd441_isr_1124828_42.stderr parsl.dataflow.dflow DEBUG: Task 187 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 187 submitted for App characterizeImage, waiting on task 186 parsl.dataflow.dflow DEBUG: Task 187 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 187 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 188 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 188 submitted for App calibrate, waiting on task 187 parsl.dataflow.dflow DEBUG: Task 188 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 188 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 189 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 189 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 189 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 189 has memoization hash 5d26217741493bebf42b1b4bdced43c2 parsl.dataflow.memoization INFO: Task 189 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adad760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 189 try 0 launched on executor multi with executor id 64 parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/f77b9c4f-8147-4020-8b41-7ccaea74586b_isr_1124707_27.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/f77b9c4f-8147-4020-8b41-7ccaea74586b_isr_1124707_27.stderr parsl.dataflow.dflow DEBUG: Task 190 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 190 submitted for App characterizeImage, waiting on task 189 parsl.dataflow.dflow DEBUG: Task 190 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 190 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 191 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 191 submitted for App calibrate, waiting on task 190 parsl.dataflow.dflow DEBUG: Task 191 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 191 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 192 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 192 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 192 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 192 has memoization hash b46afb60d4be08ef1e7ba0dbf3b7a3b0 parsl.dataflow.memoization INFO: Task 192 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adada80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 192 try 0 launched on executor multi with executor id 65 parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/385e2540-f2e2-4846-b968-21bc1c018f6b_isr_1124828_11.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/385e2540-f2e2-4846-b968-21bc1c018f6b_isr_1124828_11.stderr parsl.dataflow.dflow DEBUG: Task 193 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 193 submitted for App characterizeImage, waiting on task 192 parsl.dataflow.dflow DEBUG: Task 193 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 193 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 194 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 194 submitted for App calibrate, waiting on task 193 parsl.dataflow.dflow DEBUG: Task 194 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 194 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 195 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 195 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 195 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 195 has memoization hash 78b394c323a479ebf6408d5eb914e513 parsl.dataflow.memoization INFO: Task 195 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adadda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 195 try 0 launched on executor multi with executor id 66 parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/0f6809a8-088a-4bb8-9e3e-924bdbf0dbf2_isr_1124831_34.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/0f6809a8-088a-4bb8-9e3e-924bdbf0dbf2_isr_1124831_34.stderr parsl.dataflow.dflow DEBUG: Task 196 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 196 submitted for App characterizeImage, waiting on task 195 parsl.dataflow.dflow DEBUG: Task 196 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 196 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 197 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 197 submitted for App calibrate, waiting on task 196 parsl.dataflow.dflow DEBUG: Task 197 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 197 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 198 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 198 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 198 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 198 has memoization hash e9644eb446dba76858c3d341b2abf959 parsl.dataflow.memoization INFO: Task 198 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adae0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 198 try 0 launched on executor multi with executor id 67 parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/17d34d2b-0c31-49ed-9ede-e5f93e9f3a49_isr_1124831_53.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/17d34d2b-0c31-49ed-9ede-e5f93e9f3a49_isr_1124831_53.stderr parsl.dataflow.dflow DEBUG: Task 199 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 199 submitted for App characterizeImage, waiting on task 198 parsl.dataflow.dflow DEBUG: Task 199 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 199 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 200 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 200 submitted for App calibrate, waiting on task 199 parsl.dataflow.dflow DEBUG: Task 200 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 200 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 201 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 201 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 201 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 201 has memoization hash 1a420f731a3a05e95485740e783e2edb parsl.dataflow.memoization INFO: Task 201 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adae3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 201 try 0 launched on executor multi with executor id 68 parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/2d7f146e-31aa-445e-944d-de77039cfebd_isr_1124707_17.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/2d7f146e-31aa-445e-944d-de77039cfebd_isr_1124707_17.stderr parsl.dataflow.dflow DEBUG: Task 202 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 202 submitted for App characterizeImage, waiting on task 201 parsl.dataflow.dflow DEBUG: Task 202 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 202 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 203 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 203 submitted for App calibrate, waiting on task 202 parsl.dataflow.dflow DEBUG: Task 203 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 203 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 204 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 204 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 204 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 204 has memoization hash 943a4e3c3e932e56290d8e696186056b parsl.dataflow.memoization INFO: Task 204 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adae700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 204 try 0 launched on executor multi with executor id 69 parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/618e5d82-37ca-490a-8f10-4307b463d15b_isr_1124704_28.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/618e5d82-37ca-490a-8f10-4307b463d15b_isr_1124704_28.stderr parsl.dataflow.dflow DEBUG: Task 205 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 205 submitted for App characterizeImage, waiting on task 204 parsl.dataflow.dflow DEBUG: Task 205 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 205 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 206 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 206 submitted for App calibrate, waiting on task 205 parsl.dataflow.dflow DEBUG: Task 206 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 206 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 207 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 207 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 207 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 207 has memoization hash 2c0e91f105c6c0c1291e003512769f96 parsl.dataflow.memoization INFO: Task 207 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 207 try 0 launched on executor multi with executor id 70 parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a7e475c6-aec8-49e3-9555-259ae47f9a8c_isr_1124710_46.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a7e475c6-aec8-49e3-9555-259ae47f9a8c_isr_1124710_46.stderr parsl.dataflow.dflow DEBUG: Task 208 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 208 submitted for App characterizeImage, waiting on task 207 parsl.dataflow.dflow DEBUG: Task 208 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 208 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 209 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 209 submitted for App calibrate, waiting on task 208 parsl.dataflow.dflow DEBUG: Task 209 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 209 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 210 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 210 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 210 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 210 has memoization hash 3f8b983cd7632ff1b90b82c368a80b3d parsl.dataflow.memoization INFO: Task 210 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 210 try 0 launched on executor multi with executor id 71 parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8f43a090-1c63-4895-91d9-a55a6eff5d08_isr_1124828_9.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8f43a090-1c63-4895-91d9-a55a6eff5d08_isr_1124828_9.stderr parsl.dataflow.dflow DEBUG: Task 211 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 211 submitted for App characterizeImage, waiting on task 210 parsl.dataflow.dflow DEBUG: Task 211 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 211 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 212 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 212 submitted for App calibrate, waiting on task 211 parsl.dataflow.dflow DEBUG: Task 212 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 212 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 213 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 213 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 213 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 213 has memoization hash 9791d293404492ed1ec246cfe6a64c44 parsl.dataflow.memoization INFO: Task 213 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 213 try 0 launched on executor multi with executor id 72 parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ea008fb5-5de5-4159-8946-34a67d875ed3_isr_1124704_34.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ea008fb5-5de5-4159-8946-34a67d875ed3_isr_1124704_34.stderr parsl.dataflow.dflow DEBUG: Task 214 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 214 submitted for App characterizeImage, waiting on task 213 parsl.dataflow.dflow DEBUG: Task 214 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 214 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 215 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 215 submitted for App calibrate, waiting on task 214 parsl.dataflow.dflow DEBUG: Task 215 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 215 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 216 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 216 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 216 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 216 has memoization hash 6fc3934a14d6a41a6b21f9a0a1e82d73 parsl.dataflow.memoization INFO: Task 216 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 216 try 0 launched on executor multi with executor id 73 parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1f1b6b3e-459d-4bc3-8f8a-853c3c929bb5_isr_1124710_10.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1f1b6b3e-459d-4bc3-8f8a-853c3c929bb5_isr_1124710_10.stderr parsl.dataflow.dflow DEBUG: Task 217 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 217 submitted for App characterizeImage, waiting on task 216 parsl.dataflow.dflow DEBUG: Task 217 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 217 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 218 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 218 submitted for App calibrate, waiting on task 217 parsl.dataflow.dflow DEBUG: Task 218 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 218 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 219 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 219 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 219 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 219 has memoization hash 3a7576cfdc7ba7b59ab00fd9e320410b parsl.dataflow.memoization INFO: Task 219 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 219 try 0 launched on executor multi with executor id 74 parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/a9dc9c9d-370a-492c-9c68-ff0eac30304a_isr_1124828_17.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/a9dc9c9d-370a-492c-9c68-ff0eac30304a_isr_1124828_17.stderr parsl.dataflow.dflow DEBUG: Task 220 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 220 submitted for App characterizeImage, waiting on task 219 parsl.dataflow.dflow DEBUG: Task 220 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 220 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 221 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 221 submitted for App calibrate, waiting on task 220 parsl.dataflow.dflow DEBUG: Task 221 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 221 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 222 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 222 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 222 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 222 has memoization hash 54a813c1e8a2df9f4494f4f7124ce820 parsl.dataflow.memoization INFO: Task 222 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 222 try 0 launched on executor multi with executor id 75 parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/b6128fa6-3351-42a0-a602-7ee8c85714a8_isr_1124704_56.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/b6128fa6-3351-42a0-a602-7ee8c85714a8_isr_1124704_56.stderr parsl.dataflow.dflow DEBUG: Task 223 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 223 submitted for App characterizeImage, waiting on task 222 parsl.dataflow.dflow DEBUG: Task 223 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 223 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 224 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 224 submitted for App calibrate, waiting on task 223 parsl.dataflow.dflow DEBUG: Task 224 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 224 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 225 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 225 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 225 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 225 has memoization hash 613be797957a10f321551ef9693869a7 parsl.dataflow.memoization INFO: Task 225 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adafce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 225 try 0 launched on executor multi with executor id 76 parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/42958e77-fef6-45c1-8485-529dad11d3c3_isr_1124707_37.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/42958e77-fef6-45c1-8485-529dad11d3c3_isr_1124707_37.stderr parsl.dataflow.dflow DEBUG: Task 226 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 226 submitted for App characterizeImage, waiting on task 225 parsl.dataflow.dflow DEBUG: Task 226 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 226 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 227 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 227 submitted for App calibrate, waiting on task 226 parsl.dataflow.dflow DEBUG: Task 227 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 227 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 228 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 228 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 228 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 228 has memoization hash e19748c531dfc6ba77e334f514aa4c5b parsl.dataflow.memoization INFO: Task 228 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 228 try 0 launched on executor multi with executor id 77 parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/c3a1afdc-cd64-44ae-aaa5-89126aeb0c53_isr_1124710_50.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/c3a1afdc-cd64-44ae-aaa5-89126aeb0c53_isr_1124710_50.stderr parsl.dataflow.dflow DEBUG: Task 229 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 229 submitted for App characterizeImage, waiting on task 228 parsl.dataflow.dflow DEBUG: Task 229 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 229 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 230 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 230 submitted for App calibrate, waiting on task 229 parsl.dataflow.dflow DEBUG: Task 230 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 230 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 231 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 231 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 231 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 231 has memoization hash 680f7a94a9dd0f92ebac52a7b3e67619 parsl.dataflow.memoization INFO: Task 231 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 231 try 0 launched on executor multi with executor id 78 parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/54df688e-bbe1-4366-8b58-297140581eb2_isr_1124710_6.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/54df688e-bbe1-4366-8b58-297140581eb2_isr_1124710_6.stderr parsl.dataflow.dflow DEBUG: Task 232 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 232 submitted for App characterizeImage, waiting on task 231 parsl.dataflow.dflow DEBUG: Task 232 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 232 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 233 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 233 submitted for App calibrate, waiting on task 232 parsl.dataflow.dflow DEBUG: Task 233 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 233 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 234 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 234 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 234 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 234 has memoization hash 04b13973f481d0d8a18b35b92ca04a89 parsl.dataflow.memoization INFO: Task 234 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 234 try 0 launched on executor multi with executor id 79 parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a3231432-8848-40c1-89d5-ac1272c2c3c6_isr_1124704_43.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a3231432-8848-40c1-89d5-ac1272c2c3c6_isr_1124704_43.stderr parsl.dataflow.dflow DEBUG: Task 235 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 235 submitted for App characterizeImage, waiting on task 234 parsl.dataflow.dflow DEBUG: Task 235 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 235 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 236 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 236 submitted for App calibrate, waiting on task 235 parsl.dataflow.dflow DEBUG: Task 236 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 236 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 237 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 237 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 237 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 237 has memoization hash 97bce7cfc537a9279e44688f256ca4f8 parsl.dataflow.memoization INFO: Task 237 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 237 try 0 launched on executor multi with executor id 80 parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f2f57971-bf5d-4efc-9019-34bf927199db_isr_1124710_17.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f2f57971-bf5d-4efc-9019-34bf927199db_isr_1124710_17.stderr parsl.dataflow.dflow DEBUG: Task 238 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 238 submitted for App characterizeImage, waiting on task 237 parsl.dataflow.dflow DEBUG: Task 238 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 238 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 239 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 239 submitted for App calibrate, waiting on task 238 parsl.dataflow.dflow DEBUG: Task 239 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 239 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 240 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 240 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 240 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 240 has memoization hash 80cc014ecc47a6e567c02fe07170571b parsl.dataflow.memoization INFO: Task 240 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 240 try 0 launched on executor multi with executor id 81 parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/55858837-002b-422c-9169-fee84377d889_isr_1124704_8.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/55858837-002b-422c-9169-fee84377d889_isr_1124704_8.stderr parsl.dataflow.dflow DEBUG: Task 241 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 241 submitted for App characterizeImage, waiting on task 240 parsl.dataflow.dflow DEBUG: Task 241 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 241 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 242 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 242 submitted for App calibrate, waiting on task 241 parsl.dataflow.dflow DEBUG: Task 242 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 242 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 243 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 243 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 243 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 243 has memoization hash b81884c27c747d20181f3f7e776cec23 parsl.dataflow.memoization INFO: Task 243 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 243 try 0 launched on executor multi with executor id 82 parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/849faf6b-be4b-442b-a95b-41cae797e094_isr_1124704_27.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/849faf6b-be4b-442b-a95b-41cae797e094_isr_1124704_27.stderr parsl.dataflow.dflow DEBUG: Task 244 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 244 submitted for App characterizeImage, waiting on task 243 parsl.dataflow.dflow DEBUG: Task 244 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 244 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 245 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 245 submitted for App calibrate, waiting on task 244 parsl.dataflow.dflow DEBUG: Task 245 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 245 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 246 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 246 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 246 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 246 has memoization hash 309cd6a69e83c9826966b604c511d061 parsl.dataflow.memoization INFO: Task 246 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 246 try 0 launched on executor multi with executor id 83 parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e5db3edc-8048-4dc9-8b93-05d2056c839b_isr_1124704_40.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e5db3edc-8048-4dc9-8b93-05d2056c839b_isr_1124704_40.stderr parsl.dataflow.dflow DEBUG: Task 247 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 247 submitted for App characterizeImage, waiting on task 246 parsl.dataflow.dflow DEBUG: Task 247 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 247 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 248 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 248 submitted for App calibrate, waiting on task 247 parsl.dataflow.dflow DEBUG: Task 248 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 248 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 249 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 249 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 249 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 249 has memoization hash 3816d5fe3a273cffb9ddf3941bbca696 parsl.dataflow.memoization INFO: Task 249 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 249 try 0 launched on executor multi with executor id 84 parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/04eceea2-8bdd-4224-9ae5-c7f1c02fe902_isr_1124710_44.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/04eceea2-8bdd-4224-9ae5-c7f1c02fe902_isr_1124710_44.stderr parsl.dataflow.dflow DEBUG: Task 250 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 250 submitted for App characterizeImage, waiting on task 249 parsl.dataflow.dflow DEBUG: Task 250 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 250 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 251 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 251 submitted for App calibrate, waiting on task 250 parsl.dataflow.dflow DEBUG: Task 251 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 251 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 252 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 252 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 252 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 252 has memoization hash 090ebe02450242fe4dcc83a5f9204229 parsl.dataflow.memoization INFO: Task 252 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 252 try 0 launched on executor multi with executor id 85 parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/fb692671-4542-46e7-ab23-665b8958bab1_isr_1124704_18.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/fb692671-4542-46e7-ab23-665b8958bab1_isr_1124704_18.stderr parsl.dataflow.dflow DEBUG: Task 253 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 253 submitted for App characterizeImage, waiting on task 252 parsl.dataflow.dflow DEBUG: Task 253 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 253 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 254 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 254 submitted for App calibrate, waiting on task 253 parsl.dataflow.dflow DEBUG: Task 254 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 254 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 255 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 255 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 255 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 255 has memoization hash 5fd01e782e16118a8ee51a12fc2858b2 parsl.dataflow.memoization INFO: Task 255 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 255 try 0 launched on executor multi with executor id 86 parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/443442bb-17fb-4f4c-93ba-ca7682c7ffbc_isr_1124707_16.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/443442bb-17fb-4f4c-93ba-ca7682c7ffbc_isr_1124707_16.stderr parsl.dataflow.dflow DEBUG: Task 256 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 256 submitted for App characterizeImage, waiting on task 255 parsl.dataflow.dflow DEBUG: Task 256 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 256 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 257 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 257 submitted for App calibrate, waiting on task 256 parsl.dataflow.dflow DEBUG: Task 257 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 257 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 258 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 258 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 258 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 258 has memoization hash 1762e910b656bd5ecfd7d3ff5823d2c5 parsl.dataflow.memoization INFO: Task 258 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 258 try 0 launched on executor multi with executor id 87 parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/4ba19549-345f-4953-972a-4cc66b2c8784_isr_1124828_27.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/4ba19549-345f-4953-972a-4cc66b2c8784_isr_1124828_27.stderr parsl.dataflow.dflow DEBUG: Task 259 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 259 submitted for App characterizeImage, waiting on task 258 parsl.dataflow.dflow DEBUG: Task 259 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 259 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 260 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 260 submitted for App calibrate, waiting on task 259 parsl.dataflow.dflow DEBUG: Task 260 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 260 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 261 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 261 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 261 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 261 has memoization hash 294fa9bd9d391628e5979e41a014f017 parsl.dataflow.memoization INFO: Task 261 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aa2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 261 try 0 launched on executor multi with executor id 88 parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/e5776da1-c144-4fda-8df7-4df8b6c1fb2b_isr_1124707_10.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/e5776da1-c144-4fda-8df7-4df8b6c1fb2b_isr_1124707_10.stderr parsl.dataflow.dflow DEBUG: Task 262 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 262 submitted for App characterizeImage, waiting on task 261 parsl.dataflow.dflow DEBUG: Task 262 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 262 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 263 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 263 submitted for App calibrate, waiting on task 262 parsl.dataflow.dflow DEBUG: Task 263 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 263 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 264 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 264 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 264 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 264 has memoization hash 0ccfd45b4c38adf017b6bdefb1c77748 parsl.dataflow.memoization INFO: Task 264 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aa5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 264 try 0 launched on executor multi with executor id 89 parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/21647ca3-fc54-4d4d-84e5-b6f45589f7ca_isr_1124831_18.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/21647ca3-fc54-4d4d-84e5-b6f45589f7ca_isr_1124831_18.stderr parsl.dataflow.dflow DEBUG: Task 265 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 265 submitted for App characterizeImage, waiting on task 264 parsl.dataflow.dflow DEBUG: Task 265 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 265 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 266 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 266 submitted for App calibrate, waiting on task 265 parsl.dataflow.dflow DEBUG: Task 266 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 266 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 267 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 267 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 267 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 267 has memoization hash df88afd573a26e3015290ddb059a0f5e parsl.dataflow.memoization INFO: Task 267 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aa8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 267 try 0 launched on executor multi with executor id 90 parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/2d6cfbd3-f98e-4060-8ce1-29d8659e8f2f_isr_1124831_19.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/2d6cfbd3-f98e-4060-8ce1-29d8659e8f2f_isr_1124831_19.stderr parsl.dataflow.dflow DEBUG: Task 268 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 268 submitted for App characterizeImage, waiting on task 267 parsl.dataflow.dflow DEBUG: Task 268 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 268 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 269 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 269 submitted for App calibrate, waiting on task 268 parsl.dataflow.dflow DEBUG: Task 269 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 269 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 270 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 270 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 270 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 270 has memoization hash fba6a21a25d497eefe930001ac7f1288 parsl.dataflow.memoization INFO: Task 270 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 270 try 0 launched on executor multi with executor id 91 parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c61860d4-11cb-422e-b95b-75f4d4c5489e_isr_1124707_47.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c61860d4-11cb-422e-b95b-75f4d4c5489e_isr_1124707_47.stderr parsl.dataflow.dflow DEBUG: Task 271 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 271 submitted for App characterizeImage, waiting on task 270 parsl.dataflow.dflow DEBUG: Task 271 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 271 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 272 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 272 submitted for App calibrate, waiting on task 271 parsl.dataflow.dflow DEBUG: Task 272 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 272 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 273 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 273 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 273 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 273 has memoization hash 48e77d1784f1885b891c312700c96410 parsl.dataflow.memoization INFO: Task 273 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aaf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 273 try 0 launched on executor multi with executor id 92 parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5f47a431-def2-42f3-880f-813b5d7b729d_isr_1124707_3.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5f47a431-def2-42f3-880f-813b5d7b729d_isr_1124707_3.stderr parsl.dataflow.dflow DEBUG: Task 274 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 274 submitted for App characterizeImage, waiting on task 273 parsl.dataflow.dflow DEBUG: Task 274 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 274 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 275 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 275 submitted for App calibrate, waiting on task 274 parsl.dataflow.dflow DEBUG: Task 275 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 275 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 276 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 276 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 276 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 276 has memoization hash ec1975617fa443aa50d875c751df3ab4 parsl.dataflow.memoization INFO: Task 276 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ab240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 276 try 0 launched on executor multi with executor id 93 parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5e5abb27-b8e2-42f6-b626-cdc5f999728d_isr_1124707_24.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5e5abb27-b8e2-42f6-b626-cdc5f999728d_isr_1124707_24.stderr parsl.dataflow.dflow DEBUG: Task 277 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 277 submitted for App characterizeImage, waiting on task 276 parsl.dataflow.dflow DEBUG: Task 277 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 277 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 278 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 278 submitted for App calibrate, waiting on task 277 parsl.dataflow.dflow DEBUG: Task 278 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 278 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 279 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 279 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 279 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 279 has memoization hash e06f501ee8cf6602af5907b2ae7180e9 parsl.dataflow.memoization INFO: Task 279 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ab560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 279 try 0 launched on executor multi with executor id 94 parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/8f1fbb28-611c-42ef-9b17-11ebab374ac2_isr_1124831_52.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/8f1fbb28-611c-42ef-9b17-11ebab374ac2_isr_1124831_52.stderr parsl.dataflow.dflow DEBUG: Task 280 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 280 submitted for App characterizeImage, waiting on task 279 parsl.dataflow.dflow DEBUG: Task 280 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 280 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 281 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 281 submitted for App calibrate, waiting on task 280 parsl.dataflow.dflow DEBUG: Task 281 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 281 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 282 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 282 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 282 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 282 has memoization hash 5d9b595957b099d3a7f9167943451f6a parsl.dataflow.memoization INFO: Task 282 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ab880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 282 try 0 launched on executor multi with executor id 95 parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/aee68d42-cbd9-4eb8-b7ba-2101a2e25c75_isr_1124707_22.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/aee68d42-cbd9-4eb8-b7ba-2101a2e25c75_isr_1124707_22.stderr parsl.dataflow.dflow DEBUG: Task 283 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 283 submitted for App characterizeImage, waiting on task 282 parsl.dataflow.dflow DEBUG: Task 283 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 283 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 284 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 284 submitted for App calibrate, waiting on task 283 parsl.dataflow.dflow DEBUG: Task 284 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 284 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 285 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 285 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 285 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 285 has memoization hash 5faab2b77745f86cf6ec8d18ea0a4f5c parsl.dataflow.memoization INFO: Task 285 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297abba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 285 try 0 launched on executor multi with executor id 96 parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/65e832cd-9b4e-4086-a457-39a50581461c_isr_1124828_15.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/65e832cd-9b4e-4086-a457-39a50581461c_isr_1124828_15.stderr parsl.dataflow.dflow DEBUG: Task 286 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 286 submitted for App characterizeImage, waiting on task 285 parsl.dataflow.dflow DEBUG: Task 286 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 286 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 287 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 287 submitted for App calibrate, waiting on task 286 parsl.dataflow.dflow DEBUG: Task 287 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 287 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 288 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 288 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 288 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 288 has memoization hash 6069010e8a3cb50dac49840741cd0ef9 parsl.dataflow.memoization INFO: Task 288 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297abec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 288 try 0 launched on executor multi with executor id 97 parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/c885063a-9244-4cf8-a7cc-ec8b3f6dc276_isr_1124710_28.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/c885063a-9244-4cf8-a7cc-ec8b3f6dc276_isr_1124710_28.stderr parsl.dataflow.dflow DEBUG: Task 289 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 289 submitted for App characterizeImage, waiting on task 288 parsl.dataflow.dflow DEBUG: Task 289 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 289 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 290 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 290 submitted for App calibrate, waiting on task 289 parsl.dataflow.dflow DEBUG: Task 290 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 290 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 291 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 291 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 291 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 291 has memoization hash 7165e97dd3ad7d5908e961316175ddf1 parsl.dataflow.memoization INFO: Task 291 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cc220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 291 try 0 launched on executor multi with executor id 98 parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/aff9fb28-b99f-4f2f-a1a1-b722e6b498cc_isr_1124707_39.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/aff9fb28-b99f-4f2f-a1a1-b722e6b498cc_isr_1124707_39.stderr parsl.dataflow.dflow DEBUG: Task 292 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 292 submitted for App characterizeImage, waiting on task 291 parsl.dataflow.dflow DEBUG: Task 292 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 292 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 293 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 293 submitted for App calibrate, waiting on task 292 parsl.dataflow.dflow DEBUG: Task 293 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 293 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 294 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 294 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 294 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 294 has memoization hash 5169ea28f5074dabec5ac5e03bb77f6a parsl.dataflow.memoization INFO: Task 294 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cc540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 294 try 0 launched on executor multi with executor id 99 parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/93e46d19-5a37-4bf4-a379-94db4be1b2ad_isr_1124707_46.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/93e46d19-5a37-4bf4-a379-94db4be1b2ad_isr_1124707_46.stderr parsl.dataflow.dflow DEBUG: Task 295 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 295 submitted for App characterizeImage, waiting on task 294 parsl.dataflow.dflow DEBUG: Task 295 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 295 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 296 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 296 submitted for App calibrate, waiting on task 295 parsl.dataflow.dflow DEBUG: Task 296 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 296 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 297 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 297 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 297 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 297 has memoization hash 78df74af96ddf3bd44c71647646f834c parsl.dataflow.memoization INFO: Task 297 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cc860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 297 try 0 launched on executor multi with executor id 100 parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/eb36a66d-ef2d-480c-99e5-327ef2e171d9_isr_1124828_43.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/eb36a66d-ef2d-480c-99e5-327ef2e171d9_isr_1124828_43.stderr parsl.dataflow.dflow DEBUG: Task 298 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 298 submitted for App characterizeImage, waiting on task 297 parsl.dataflow.dflow DEBUG: Task 298 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 298 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 299 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 299 submitted for App calibrate, waiting on task 298 parsl.dataflow.dflow DEBUG: Task 299 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 299 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 300 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 300 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 300 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 300 has memoization hash b78971599edc05eab4a4044d5c172cea parsl.dataflow.memoization INFO: Task 300 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ccb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 300 try 0 launched on executor multi with executor id 101 parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5868be83-b43f-45b2-b32e-8b3a424ed4e3_isr_1124707_43.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5868be83-b43f-45b2-b32e-8b3a424ed4e3_isr_1124707_43.stderr parsl.dataflow.dflow DEBUG: Task 301 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 301 submitted for App characterizeImage, waiting on task 300 parsl.dataflow.dflow DEBUG: Task 301 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 301 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 302 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 302 submitted for App calibrate, waiting on task 301 parsl.dataflow.dflow DEBUG: Task 302 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 302 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 303 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 303 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 303 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 303 has memoization hash 9f1c533ac9b38b9f79813699e77f5732 parsl.dataflow.memoization INFO: Task 303 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ccea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 303 try 0 launched on executor multi with executor id 102 parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/63fb24f9-bf4a-403f-9e77-348988cb3880_isr_1124831_14.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/63fb24f9-bf4a-403f-9e77-348988cb3880_isr_1124831_14.stderr parsl.dataflow.dflow DEBUG: Task 304 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 304 submitted for App characterizeImage, waiting on task 303 parsl.dataflow.dflow DEBUG: Task 304 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 304 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 305 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 305 submitted for App calibrate, waiting on task 304 parsl.dataflow.dflow DEBUG: Task 305 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 305 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 306 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 306 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 306 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 306 has memoization hash c2c678c8bf9ba7ccfb29403215ba7564 parsl.dataflow.memoization INFO: Task 306 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cd1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 306 try 0 launched on executor multi with executor id 103 parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/2295dd8a-1684-4c1b-b973-d468334901b0_isr_1124707_60.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/2295dd8a-1684-4c1b-b973-d468334901b0_isr_1124707_60.stderr parsl.dataflow.dflow DEBUG: Task 307 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 307 submitted for App characterizeImage, waiting on task 306 parsl.dataflow.dflow DEBUG: Task 307 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 307 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 308 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 308 submitted for App calibrate, waiting on task 307 parsl.dataflow.dflow DEBUG: Task 308 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 308 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 309 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 309 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 309 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 309 has memoization hash 7b0fc5cd4574fba5db424776a162160d parsl.dataflow.memoization INFO: Task 309 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cd4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 309 try 0 launched on executor multi with executor id 104 parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6a057159-5fc8-4bbf-984b-581889166c82_isr_1124704_39.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6a057159-5fc8-4bbf-984b-581889166c82_isr_1124704_39.stderr parsl.dataflow.dflow DEBUG: Task 310 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 310 submitted for App characterizeImage, waiting on task 309 parsl.dataflow.dflow DEBUG: Task 310 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 310 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 311 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 311 submitted for App calibrate, waiting on task 310 parsl.dataflow.dflow DEBUG: Task 311 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 311 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 312 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 312 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 312 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 312 has memoization hash 9ec6a0b9cd91ac3ebc227c76565cc66c parsl.dataflow.memoization INFO: Task 312 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cd800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 312 try 0 launched on executor multi with executor id 105 parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/3cc4896d-3102-4af8-88eb-34a474177ac6_isr_1124831_1.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/3cc4896d-3102-4af8-88eb-34a474177ac6_isr_1124831_1.stderr parsl.dataflow.dflow DEBUG: Task 313 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 313 submitted for App characterizeImage, waiting on task 312 parsl.dataflow.dflow DEBUG: Task 313 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 313 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 314 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 314 submitted for App calibrate, waiting on task 313 parsl.dataflow.dflow DEBUG: Task 314 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 314 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 315 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 315 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 315 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 315 has memoization hash 43e9d6c7a32455429ce108645bda0892 parsl.dataflow.memoization INFO: Task 315 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cdb20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 315 try 0 launched on executor multi with executor id 106 parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/2b1d41c8-d34f-4b3a-9c87-f1d50badb8c0_isr_1124831_57.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/2b1d41c8-d34f-4b3a-9c87-f1d50badb8c0_isr_1124831_57.stderr parsl.dataflow.dflow DEBUG: Task 316 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 316 submitted for App characterizeImage, waiting on task 315 parsl.dataflow.dflow DEBUG: Task 316 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 316 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 317 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 317 submitted for App calibrate, waiting on task 316 parsl.dataflow.dflow DEBUG: Task 317 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 317 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 318 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 318 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 318 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 318 has memoization hash 590c855f8f5721426e01083764f7d044 parsl.dataflow.memoization INFO: Task 318 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cde40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 318 try 0 launched on executor multi with executor id 107 parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/a8359e7d-1cca-41a9-ac0d-a13299011532_isr_1124828_18.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/a8359e7d-1cca-41a9-ac0d-a13299011532_isr_1124828_18.stderr parsl.dataflow.dflow DEBUG: Task 319 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 319 submitted for App characterizeImage, waiting on task 318 parsl.dataflow.dflow DEBUG: Task 319 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 319 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 320 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 320 submitted for App calibrate, waiting on task 319 parsl.dataflow.dflow DEBUG: Task 320 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 320 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 321 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 321 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 321 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 321 has memoization hash fe5b1512ea47a73c26abc553e864b3b6 parsl.dataflow.memoization INFO: Task 321 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ce160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 321 try 0 launched on executor multi with executor id 108 parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/523d6d35-6cca-4762-9ab3-5a9e160f3790_isr_1124831_36.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/523d6d35-6cca-4762-9ab3-5a9e160f3790_isr_1124831_36.stderr parsl.dataflow.dflow DEBUG: Task 322 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 322 submitted for App characterizeImage, waiting on task 321 parsl.dataflow.dflow DEBUG: Task 322 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 322 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 323 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 323 submitted for App calibrate, waiting on task 322 parsl.dataflow.dflow DEBUG: Task 323 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 323 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 324 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 324 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 324 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 324 has memoization hash f690692f672b024995bec6a09f562c82 parsl.dataflow.memoization INFO: Task 324 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ce480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 324 try 0 launched on executor multi with executor id 109 parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/149ac961-b2a1-4d5a-840e-b93ebf78cfeb_isr_1124710_15.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/149ac961-b2a1-4d5a-840e-b93ebf78cfeb_isr_1124710_15.stderr parsl.dataflow.dflow DEBUG: Task 325 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 325 submitted for App characterizeImage, waiting on task 324 parsl.dataflow.dflow DEBUG: Task 325 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 325 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 326 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 326 submitted for App calibrate, waiting on task 325 parsl.dataflow.dflow DEBUG: Task 326 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 326 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 327 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 327 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 327 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 327 has memoization hash 2c3b990ffdbae059bcc372913fb487e4 parsl.dataflow.memoization INFO: Task 327 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ce7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 327 try 0 launched on executor multi with executor id 110 parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/41401686-4a91-4f0d-94cb-e4c7fa4fea4c_isr_1124828_60.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/41401686-4a91-4f0d-94cb-e4c7fa4fea4c_isr_1124828_60.stderr parsl.dataflow.dflow DEBUG: Task 328 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 328 submitted for App characterizeImage, waiting on task 327 parsl.dataflow.dflow DEBUG: Task 328 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 328 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 329 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 329 submitted for App calibrate, waiting on task 328 parsl.dataflow.dflow DEBUG: Task 329 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 329 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 330 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 330 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 330 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 330 has memoization hash 85b51c206e49952d65cb86503dd594ae parsl.dataflow.memoization INFO: Task 330 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ceac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 330 try 0 launched on executor multi with executor id 111 parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/b419e3c1-e106-4696-a283-4cb8ab611de8_isr_1124704_12.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/b419e3c1-e106-4696-a283-4cb8ab611de8_isr_1124704_12.stderr parsl.dataflow.dflow DEBUG: Task 331 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 331 submitted for App characterizeImage, waiting on task 330 parsl.dataflow.dflow DEBUG: Task 331 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 331 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 332 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 332 submitted for App calibrate, waiting on task 331 parsl.dataflow.dflow DEBUG: Task 332 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 332 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 333 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 333 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 333 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 333 has memoization hash c0f4ec15b644b3f552c1e969fc8079bd parsl.dataflow.memoization INFO: Task 333 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 333 try 0 launched on executor multi with executor id 112 parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/aa33f37f-ea1e-416b-b7fe-935f87690744_isr_1124831_43.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/aa33f37f-ea1e-416b-b7fe-935f87690744_isr_1124831_43.stderr parsl.dataflow.dflow DEBUG: Task 334 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 334 submitted for App characterizeImage, waiting on task 333 parsl.dataflow.dflow DEBUG: Task 334 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 334 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 335 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 335 submitted for App calibrate, waiting on task 334 parsl.dataflow.dflow DEBUG: Task 335 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 335 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 336 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 336 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 336 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 336 has memoization hash 78ef158ed8446abad50317c6ce798716 parsl.dataflow.memoization INFO: Task 336 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cf100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 336 try 0 launched on executor multi with executor id 113 parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/7bf17bef-d950-4c74-8b96-21d00264970c_isr_1124828_31.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/7bf17bef-d950-4c74-8b96-21d00264970c_isr_1124828_31.stderr parsl.dataflow.dflow DEBUG: Task 337 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 337 submitted for App characterizeImage, waiting on task 336 parsl.dataflow.dflow DEBUG: Task 337 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 337 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 338 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 338 submitted for App calibrate, waiting on task 337 parsl.dataflow.dflow DEBUG: Task 338 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 338 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 339 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 339 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 339 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 339 has memoization hash 1da921fd3c2c8f39a096648c86c03f2f parsl.dataflow.memoization INFO: Task 339 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cf420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 339 try 0 launched on executor multi with executor id 114 parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c6cb0395-fbd2-49c3-835b-d9550b64ac83_isr_1124707_34.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c6cb0395-fbd2-49c3-835b-d9550b64ac83_isr_1124707_34.stderr parsl.dataflow.dflow DEBUG: Task 340 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 340 submitted for App characterizeImage, waiting on task 339 parsl.dataflow.dflow DEBUG: Task 340 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 340 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 341 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 341 submitted for App calibrate, waiting on task 340 parsl.dataflow.dflow DEBUG: Task 341 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 341 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 342 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 342 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 342 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 342 has memoization hash 3fc75d06b67a524381ea644bc83ab31f parsl.dataflow.memoization INFO: Task 342 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cf740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 342 try 0 launched on executor multi with executor id 115 parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/b4434d0a-a53e-4d09-820e-f8f2070faa58_isr_1124707_38.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/b4434d0a-a53e-4d09-820e-f8f2070faa58_isr_1124707_38.stderr parsl.dataflow.dflow DEBUG: Task 343 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 343 submitted for App characterizeImage, waiting on task 342 parsl.dataflow.dflow DEBUG: Task 343 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 343 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 344 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 344 submitted for App calibrate, waiting on task 343 parsl.dataflow.dflow DEBUG: Task 344 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 344 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 345 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 345 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 345 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 345 has memoization hash 8bd5267ac99bdd07f811fa17e9b68f6c parsl.dataflow.memoization INFO: Task 345 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cfa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 345 try 0 launched on executor multi with executor id 116 parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/ee318162-fa33-47b7-a8f9-9c886a4dacdf_isr_1124710_27.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/ee318162-fa33-47b7-a8f9-9c886a4dacdf_isr_1124710_27.stderr parsl.dataflow.dflow DEBUG: Task 346 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 346 submitted for App characterizeImage, waiting on task 345 parsl.dataflow.dflow DEBUG: Task 346 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 346 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 347 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 347 submitted for App calibrate, waiting on task 346 parsl.dataflow.dflow DEBUG: Task 347 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 347 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 348 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 348 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 348 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 348 has memoization hash 98670f06b74e37e12cec5d3f0c27f5f6 parsl.dataflow.memoization INFO: Task 348 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cfd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 348 try 0 launched on executor multi with executor id 117 parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e2f6b261-c1a1-461e-8c67-26b8085bf9af_isr_1124704_20.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e2f6b261-c1a1-461e-8c67-26b8085bf9af_isr_1124704_20.stderr parsl.dataflow.dflow DEBUG: Task 349 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 349 submitted for App characterizeImage, waiting on task 348 parsl.dataflow.dflow DEBUG: Task 349 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 349 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 350 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 350 submitted for App calibrate, waiting on task 349 parsl.dataflow.dflow DEBUG: Task 350 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 350 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 351 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 351 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 351 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 351 has memoization hash d00b3934069d5b7b3f4409e40d5fd71a parsl.dataflow.memoization INFO: Task 351 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 351 try 0 launched on executor multi with executor id 118 parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/c2d84418-9f1c-49da-b688-2817a0c11bda_isr_1124704_4.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/c2d84418-9f1c-49da-b688-2817a0c11bda_isr_1124704_4.stderr parsl.dataflow.dflow DEBUG: Task 352 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 352 submitted for App characterizeImage, waiting on task 351 parsl.dataflow.dflow DEBUG: Task 352 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 352 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 353 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 353 submitted for App calibrate, waiting on task 352 parsl.dataflow.dflow DEBUG: Task 353 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 353 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 354 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 354 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 354 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 354 has memoization hash 46773e63652dfa51c07d74c360e77361 parsl.dataflow.memoization INFO: Task 354 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 354 try 0 launched on executor multi with executor id 119 parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/0cb09ee3-3852-4740-94f4-8edec67fa0de_isr_1124828_29.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/0cb09ee3-3852-4740-94f4-8edec67fa0de_isr_1124828_29.stderr parsl.dataflow.dflow DEBUG: Task 355 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 355 submitted for App characterizeImage, waiting on task 354 parsl.dataflow.dflow DEBUG: Task 355 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 355 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 356 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 356 submitted for App calibrate, waiting on task 355 parsl.dataflow.dflow DEBUG: Task 356 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 356 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 357 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 357 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 357 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 357 has memoization hash e774147670cd6ee6f0322f6da9cc4274 parsl.dataflow.memoization INFO: Task 357 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 357 try 0 launched on executor multi with executor id 120 parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/2d631d3f-e902-4037-878c-40d7b38dd921_isr_1124710_43.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/2d631d3f-e902-4037-878c-40d7b38dd921_isr_1124710_43.stderr parsl.dataflow.dflow DEBUG: Task 358 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 358 submitted for App characterizeImage, waiting on task 357 parsl.dataflow.dflow DEBUG: Task 358 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 358 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 359 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 359 submitted for App calibrate, waiting on task 358 parsl.dataflow.dflow DEBUG: Task 359 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 359 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 360 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 360 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 360 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 360 has memoization hash 2f14c129a57bd76ac8627266b55319d9 parsl.dataflow.memoization INFO: Task 360 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 360 try 0 launched on executor multi with executor id 121 parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/a4185c30-5c0a-4899-b03c-f9b57a8955fd_isr_1124707_8.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/a4185c30-5c0a-4899-b03c-f9b57a8955fd_isr_1124707_8.stderr parsl.dataflow.dflow DEBUG: Task 361 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 361 submitted for App characterizeImage, waiting on task 360 parsl.dataflow.dflow DEBUG: Task 361 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 361 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 362 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 362 submitted for App calibrate, waiting on task 361 parsl.dataflow.dflow DEBUG: Task 362 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 362 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 363 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 363 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 363 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 363 has memoization hash ae5666296d24ec18d9d57b6539df5de2 parsl.dataflow.memoization INFO: Task 363 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 363 try 0 launched on executor multi with executor id 122 parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/9dce3d7c-43ce-45c7-b5b2-5c679efb2c9f_isr_1124704_19.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/9dce3d7c-43ce-45c7-b5b2-5c679efb2c9f_isr_1124704_19.stderr parsl.dataflow.dflow DEBUG: Task 364 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 364 submitted for App characterizeImage, waiting on task 363 parsl.dataflow.dflow DEBUG: Task 364 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 364 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 365 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 365 submitted for App calibrate, waiting on task 364 parsl.dataflow.dflow DEBUG: Task 365 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 365 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 366 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 366 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 366 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 366 has memoization hash 5dfc8620607255760ed67f3f0d422a1e parsl.dataflow.memoization INFO: Task 366 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 366 try 0 launched on executor multi with executor id 123 parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/8029bcb4-6279-419e-b037-19b095ef0e14_isr_1124710_26.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/8029bcb4-6279-419e-b037-19b095ef0e14_isr_1124710_26.stderr parsl.dataflow.dflow DEBUG: Task 367 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 367 submitted for App characterizeImage, waiting on task 366 parsl.dataflow.dflow DEBUG: Task 367 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 367 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 368 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 368 submitted for App calibrate, waiting on task 367 parsl.dataflow.dflow DEBUG: Task 368 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 368 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 369 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 369 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 369 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 369 has memoization hash 4fab9ded05f97de7469eed738e2bbf28 parsl.dataflow.memoization INFO: Task 369 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e93a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 369 try 0 launched on executor multi with executor id 124 parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/469ffe4a-a533-4333-bac3-e6ba2009c98b_isr_1124831_61.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/469ffe4a-a533-4333-bac3-e6ba2009c98b_isr_1124831_61.stderr parsl.dataflow.dflow DEBUG: Task 370 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 370 submitted for App characterizeImage, waiting on task 369 parsl.dataflow.dflow DEBUG: Task 370 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 370 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 371 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 371 submitted for App calibrate, waiting on task 370 parsl.dataflow.dflow DEBUG: Task 371 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 371 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 372 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 372 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 372 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 372 has memoization hash c83289bb7b69a75a94ea56d1751950fa parsl.dataflow.memoization INFO: Task 372 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 372 try 0 launched on executor multi with executor id 125 parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/9afd980d-28be-422b-858e-31b2734659bc_isr_1124831_29.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/9afd980d-28be-422b-858e-31b2734659bc_isr_1124831_29.stderr parsl.dataflow.dflow DEBUG: Task 373 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 373 submitted for App characterizeImage, waiting on task 372 parsl.dataflow.dflow DEBUG: Task 373 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 373 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 374 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 374 submitted for App calibrate, waiting on task 373 parsl.dataflow.dflow DEBUG: Task 374 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 374 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 375 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 375 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 375 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 375 has memoization hash d7c5f9e0e2257f2b3d61ccb300208782 parsl.dataflow.memoization INFO: Task 375 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 07:25:40 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local parsl.dataflow.dflow INFO: Parsl task 375 try 0 launched on executor multi with executor id 126 parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/b8d4d2a9-d54a-4dfa-ae41-02010fed3343_isr_1124704_26.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/b8d4d2a9-d54a-4dfa-ae41-02010fed3343_isr_1124704_26.stderr parsl.dataflow.dflow DEBUG: Task 376 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 376 submitted for App characterizeImage, waiting on task 375 parsl.dataflow.dflow DEBUG: Task 376 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 376 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 377 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 377 submitted for App calibrate, waiting on task 376 parsl.dataflow.dflow DEBUG: Task 377 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 377 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 378 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 378 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 378 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 378 has memoization hash 8e297212bddc52d34bb605411ed853c8 parsl.dataflow.memoization INFO: Task 378 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ebd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 378 try 0 launched on executor multi with executor id 127 parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/1464ff7b-dde9-4e3c-ab63-905e3326595e_isr_1124704_45.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/1464ff7b-dde9-4e3c-ab63-905e3326595e_isr_1124704_45.stderr parsl.dataflow.dflow DEBUG: Task 379 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 379 submitted for App characterizeImage, waiting on task 378 parsl.dataflow.dflow DEBUG: Task 379 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 379 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 380 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 380 submitted for App calibrate, waiting on task 379 parsl.dataflow.dflow DEBUG: Task 380 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 380 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 381 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 381 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 381 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 381 has memoization hash 2e0f01160cde06eb3dfd2569e1d39600 parsl.dataflow.memoization INFO: Task 381 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.dflow INFO: Parsl task 381 try 0 launched on executor multi with executor id 128 parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/7d10d2ae-4399-4db2-b4a9-fefb0f96d598_isr_1124710_51.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/7d10d2ae-4399-4db2-b4a9-fefb0f96d598_isr_1124710_51.stderr parsl.dataflow.dflow DEBUG: Task 382 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 382 submitted for App characterizeImage, waiting on task 381 parsl.dataflow.dflow DEBUG: Task 382 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 382 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 383 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 383 submitted for App calibrate, waiting on task 382 parsl.dataflow.dflow DEBUG: Task 383 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 383 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 384 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 384 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 384 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 384 has memoization hash 6b908be95ee05f6891e8a961e640f402 parsl.dataflow.memoization INFO: Task 384 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ea020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 1, active_tasks = 128 parsl.dataflow.strategy DEBUG: Executor multi has 128 active tasks, 1/0 running/pending blocks, and 2 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 1 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 31 more blocks parsl.executors.status_handling INFO: Scaling out by 31 blocks parsl.executors.status_handling INFO: Allocated block ID 1 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-1 2024-03-07 07:25:40 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-1 2024-03-07 07:25:40 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:40 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:40 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 07:25:40 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:40 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=1 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 384 try 0 launched on executor multi with executor id 129 parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/a900e9de-7224-449a-a5a0-482e817dce4f_isr_1124828_48.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/a900e9de-7224-449a-a5a0-482e817dce4f_isr_1124828_48.stderr parsl.dataflow.dflow DEBUG: Task 385 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 385 submitted for App characterizeImage, waiting on task 384 parsl.dataflow.dflow DEBUG: Task 385 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 385 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 386 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 386 submitted for App calibrate, waiting on task 385 parsl.dataflow.dflow DEBUG: Task 386 set to pending state with AppFuture: parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 386 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 387 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 387 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 387 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 387 has memoization hash 8ca0662438a2ef996ebf8e76258a7fbe parsl.dataflow.memoization INFO: Task 387 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ea5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 387 try 0 launched on executor multi with executor id 130 parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6566e522-7234-4c9e-b329-fdb4ef392647_isr_1124704_7.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6566e522-7234-4c9e-b329-fdb4ef392647_isr_1124704_7.stderr parsl.dataflow.dflow DEBUG: Task 388 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 388 submitted for App characterizeImage, waiting on task 387 parsl.dataflow.dflow DEBUG: Task 388 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 388 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 389 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 389 submitted for App calibrate, waiting on task 388 parsl.dataflow.dflow DEBUG: Task 389 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 389 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 390 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 390 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 390 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 390 has memoization hash f6bc097a40f2c693e575d93debb33f3d parsl.dataflow.memoization INFO: Task 390 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 390 try 0 launched on executor multi with executor id 131 parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/e3eef410-6b4a-439c-86d3-69d761f24e45_isr_1124828_30.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/e3eef410-6b4a-439c-86d3-69d761f24e45_isr_1124828_30.stderr parsl.dataflow.dflow DEBUG: Task 391 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 391 submitted for App characterizeImage, waiting on task 390 parsl.dataflow.dflow DEBUG: Task 391 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 391 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 392 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 392 submitted for App calibrate, waiting on task 391 parsl.dataflow.dflow DEBUG: Task 392 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 392 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 393 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 393 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 393 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 393 has memoization hash 183b038a804b518aa746eb0381b5d923 parsl.dataflow.memoization INFO: Task 393 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 393 try 0 launched on executor multi with executor id 132 parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/efee96ae-33ba-447a-8a39-0be85d628f52_isr_1124710_48.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/efee96ae-33ba-447a-8a39-0be85d628f52_isr_1124710_48.stderr parsl.dataflow.dflow DEBUG: Task 394 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 394 submitted for App characterizeImage, waiting on task 393 parsl.dataflow.dflow DEBUG: Task 394 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 394 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 395 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 395 submitted for App calibrate, waiting on task 394 parsl.dataflow.dflow DEBUG: Task 395 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 395 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 396 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 396 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 396 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 396 has memoization hash 4bcf22379b32637d1f9ebf199d3c2d5f parsl.dataflow.memoization INFO: Task 396 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 396 try 0 launched on executor multi with executor id 133 parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/d15c45b2-66be-46d5-b098-a324f0293c85_isr_1124828_46.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/d15c45b2-66be-46d5-b098-a324f0293c85_isr_1124828_46.stderr parsl.dataflow.dflow DEBUG: Task 397 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 397 submitted for App characterizeImage, waiting on task 396 parsl.dataflow.dflow DEBUG: Task 397 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 397 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 398 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 398 submitted for App calibrate, waiting on task 397 parsl.dataflow.dflow DEBUG: Task 398 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 398 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 399 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 399 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 399 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 399 has memoization hash f774b27eb14e91e0845e39896b837e13 parsl.dataflow.memoization INFO: Task 399 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 399 try 0 launched on executor multi with executor id 134 parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ba9d6043-a928-4dc2-a4cd-b0148deefd24_isr_1124704_46.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ba9d6043-a928-4dc2-a4cd-b0148deefd24_isr_1124704_46.stderr parsl.dataflow.dflow DEBUG: Task 400 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 400 submitted for App characterizeImage, waiting on task 399 parsl.dataflow.dflow DEBUG: Task 400 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 400 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 401 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 401 submitted for App calibrate, waiting on task 400 parsl.dataflow.dflow DEBUG: Task 401 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 401 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 402 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 402 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 402 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 402 has memoization hash 63857381bcb84ade73970f53a78d0a2c parsl.dataflow.memoization INFO: Task 402 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 402 try 0 launched on executor multi with executor id 135 parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/a24ef897-9c7b-4b24-ad33-085ac37d5a1b_isr_1124707_15.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/a24ef897-9c7b-4b24-ad33-085ac37d5a1b_isr_1124707_15.stderr parsl.dataflow.dflow DEBUG: Task 403 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 403 submitted for App characterizeImage, waiting on task 402 parsl.dataflow.dflow DEBUG: Task 403 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 403 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 404 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 404 submitted for App calibrate, waiting on task 403 parsl.dataflow.dflow DEBUG: Task 404 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 404 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 405 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 405 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 405 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 405 has memoization hash 9f4f38c250ec1bb3f71089d241ae8d98 parsl.dataflow.memoization INFO: Task 405 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 405 try 0 launched on executor multi with executor id 136 parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8338e9cd-d266-4aae-a3b0-19dd92053dd6_isr_1124828_37.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8338e9cd-d266-4aae-a3b0-19dd92053dd6_isr_1124828_37.stderr parsl.dataflow.dflow DEBUG: Task 406 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 406 submitted for App characterizeImage, waiting on task 405 parsl.dataflow.dflow DEBUG: Task 406 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 406 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 407 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 407 submitted for App calibrate, waiting on task 406 parsl.dataflow.dflow DEBUG: Task 407 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 407 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 408 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 408 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 408 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 408 has memoization hash f84bf233c652f198448419ea90cddfb9 parsl.dataflow.memoization INFO: Task 408 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 408 try 0 launched on executor multi with executor id 137 parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/de53283b-1af0-4ab8-b87b-8c1b68f01d7f_isr_1124707_48.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/de53283b-1af0-4ab8-b87b-8c1b68f01d7f_isr_1124707_48.stderr parsl.dataflow.dflow DEBUG: Task 409 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 409 submitted for App characterizeImage, waiting on task 408 parsl.dataflow.dflow DEBUG: Task 409 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 409 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 410 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 410 submitted for App calibrate, waiting on task 409 parsl.dataflow.dflow DEBUG: Task 410 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 410 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 411 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 411 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 411 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 411 has memoization hash cac550fe12448aca753b09dec8ebbeb9 parsl.dataflow.memoization INFO: Task 411 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 411 try 0 launched on executor multi with executor id 138 parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/397495d7-aa4b-49a7-a084-4a117af34e36_isr_1124707_5.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/397495d7-aa4b-49a7-a084-4a117af34e36_isr_1124707_5.stderr parsl.dataflow.dflow DEBUG: Task 412 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 412 submitted for App characterizeImage, waiting on task 411 parsl.dataflow.dflow DEBUG: Task 412 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 412 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 413 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 413 submitted for App calibrate, waiting on task 412 parsl.dataflow.dflow DEBUG: Task 413 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 413 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 414 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 414 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 414 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 414 has memoization hash a694b0ea574cae329ac9e3d1eac5cf88 parsl.dataflow.memoization INFO: Task 414 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ebc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 414 try 0 launched on executor multi with executor id 139 parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/69dc5d70-3b7f-4197-9e5f-f812aa85a6ac_isr_1124707_9.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/69dc5d70-3b7f-4197-9e5f-f812aa85a6ac_isr_1124707_9.stderr parsl.dataflow.dflow DEBUG: Task 415 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 415 submitted for App characterizeImage, waiting on task 414 parsl.dataflow.dflow DEBUG: Task 415 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 415 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 416 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 416 submitted for App calibrate, waiting on task 415 parsl.dataflow.dflow DEBUG: Task 416 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 416 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 417 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 417 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 417 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 417 has memoization hash 71191b644a7f5c5025b6ae90159ed504 parsl.dataflow.memoization INFO: Task 417 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 417 try 0 launched on executor multi with executor id 140 parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/d1f2a26c-4b56-4a8a-958a-43f5553a7700_isr_1124710_53.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/d1f2a26c-4b56-4a8a-958a-43f5553a7700_isr_1124710_53.stderr parsl.dataflow.dflow DEBUG: Task 418 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 418 submitted for App characterizeImage, waiting on task 417 parsl.dataflow.dflow DEBUG: Task 418 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 418 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 419 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 419 submitted for App calibrate, waiting on task 418 parsl.dataflow.dflow DEBUG: Task 419 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 419 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 420 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 420 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 420 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 420 has memoization hash ddca065e0496b1c902f7c3b8f7c76754 parsl.dataflow.memoization INFO: Task 420 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 420 try 0 launched on executor multi with executor id 141 parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/3dd8b0e8-e8db-4311-907e-63039da12afd_isr_1124828_16.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/3dd8b0e8-e8db-4311-907e-63039da12afd_isr_1124828_16.stderr parsl.dataflow.dflow DEBUG: Task 421 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 421 submitted for App characterizeImage, waiting on task 420 parsl.dataflow.dflow DEBUG: Task 421 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 421 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 422 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 422 submitted for App calibrate, waiting on task 421 parsl.dataflow.dflow DEBUG: Task 422 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 422 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 423 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 423 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 423 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 423 has memoization hash cc70faa7b92b1e233037ad1a5f035202 parsl.dataflow.memoization INFO: Task 423 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 423 try 0 launched on executor multi with executor id 142 parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e172a54d-5269-415b-9717-68fb36974086_isr_1124704_41.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e172a54d-5269-415b-9717-68fb36974086_isr_1124704_41.stderr parsl.dataflow.dflow DEBUG: Task 424 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 424 submitted for App characterizeImage, waiting on task 423 parsl.dataflow.dflow DEBUG: Task 424 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 424 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 425 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 425 submitted for App calibrate, waiting on task 424 parsl.dataflow.dflow DEBUG: Task 425 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 425 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 426 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 426 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 426 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 426 has memoization hash 24f1088288399d51f087fd33e4e138a0 parsl.dataflow.memoization INFO: Task 426 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 426 try 0 launched on executor multi with executor id 143 parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/e4ab1bcb-25f3-46a0-b1a7-e9540f54665d_isr_1124831_60.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/e4ab1bcb-25f3-46a0-b1a7-e9540f54665d_isr_1124831_60.stderr parsl.dataflow.dflow DEBUG: Task 427 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 427 submitted for App characterizeImage, waiting on task 426 parsl.dataflow.dflow DEBUG: Task 427 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 427 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 428 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 428 submitted for App calibrate, waiting on task 427 parsl.dataflow.dflow DEBUG: Task 428 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 428 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 429 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 429 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 429 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 429 has memoization hash cf449ea4e94479c89074f7c3404b09a8 parsl.dataflow.memoization INFO: Task 429 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 429 try 0 launched on executor multi with executor id 144 parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/9b9db11d-5414-4e5c-8cac-12f3ebafb89b_isr_1124707_42.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/9b9db11d-5414-4e5c-8cac-12f3ebafb89b_isr_1124707_42.stderr parsl.dataflow.dflow DEBUG: Task 430 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 430 submitted for App characterizeImage, waiting on task 429 parsl.dataflow.dflow DEBUG: Task 430 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 430 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 431 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 431 submitted for App calibrate, waiting on task 430 parsl.dataflow.dflow DEBUG: Task 431 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 431 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 432 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 432 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 432 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 432 has memoization hash 81f08a7ed3d8259fdb5fae7492e5aa4a parsl.dataflow.memoization INFO: Task 432 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 432 try 0 launched on executor multi with executor id 145 parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8df0d3b8-6710-443d-8726-95db9409d197_isr_1124828_36.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8df0d3b8-6710-443d-8726-95db9409d197_isr_1124828_36.stderr parsl.dataflow.dflow DEBUG: Task 433 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 433 submitted for App characterizeImage, waiting on task 432 parsl.dataflow.dflow DEBUG: Task 433 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 433 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 434 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 434 submitted for App calibrate, waiting on task 433 parsl.dataflow.dflow DEBUG: Task 434 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 434 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 435 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 435 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 435 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 435 has memoization hash 654131854e800e1c7fa0799f5ee8d702 parsl.dataflow.memoization INFO: Task 435 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 435 try 0 launched on executor multi with executor id 146 parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/0ffaa047-75c0-4815-809c-ee1a2d30e2fa_isr_1124831_44.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/0ffaa047-75c0-4815-809c-ee1a2d30e2fa_isr_1124831_44.stderr parsl.dataflow.dflow DEBUG: Task 436 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 436 submitted for App characterizeImage, waiting on task 435 parsl.dataflow.dflow DEBUG: Task 436 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 436 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 437 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 437 submitted for App calibrate, waiting on task 436 parsl.dataflow.dflow DEBUG: Task 437 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 437 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 438 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 438 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 438 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 438 has memoization hash ac14081fff13aa487c909a6a8748fea2 parsl.dataflow.memoization INFO: Task 438 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 438 try 0 launched on executor multi with executor id 147 parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/4119fea4-5196-4ae2-b2f1-a71a9bcfcc86_isr_1124828_23.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/4119fea4-5196-4ae2-b2f1-a71a9bcfcc86_isr_1124828_23.stderr parsl.dataflow.dflow DEBUG: Task 439 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 439 submitted for App characterizeImage, waiting on task 438 parsl.dataflow.dflow DEBUG: Task 439 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 439 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 440 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 440 submitted for App calibrate, waiting on task 439 parsl.dataflow.dflow DEBUG: Task 440 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 440 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 441 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 441 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 441 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 441 has memoization hash f1de8a5a4b291c451e4f9438a2b50a59 parsl.dataflow.memoization INFO: Task 441 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 441 try 0 launched on executor multi with executor id 148 parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ce5d6539-ec19-4f97-86f0-f55331c83ec1_isr_1124704_10.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ce5d6539-ec19-4f97-86f0-f55331c83ec1_isr_1124704_10.stderr parsl.dataflow.dflow DEBUG: Task 442 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 442 submitted for App characterizeImage, waiting on task 441 parsl.dataflow.dflow DEBUG: Task 442 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 442 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 443 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 443 submitted for App calibrate, waiting on task 442 parsl.dataflow.dflow DEBUG: Task 443 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 443 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 444 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 444 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 444 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 444 has memoization hash e73560942072fdc530ad0f49396cd5c4 parsl.dataflow.memoization INFO: Task 444 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 444 try 0 launched on executor multi with executor id 149 parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/76c2964e-4bd2-4613-9c6f-48b8b78c8af8_isr_1124828_45.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/76c2964e-4bd2-4613-9c6f-48b8b78c8af8_isr_1124828_45.stderr parsl.dataflow.dflow DEBUG: Task 445 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 445 submitted for App characterizeImage, waiting on task 444 parsl.dataflow.dflow DEBUG: Task 445 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 445 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 446 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 446 submitted for App calibrate, waiting on task 445 parsl.dataflow.dflow DEBUG: Task 446 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 446 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 447 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 447 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 447 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 447 has memoization hash 5f7e237e0d22e5b9d0c65dfe3ae81c21 parsl.dataflow.memoization INFO: Task 447 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 07:25:41 proc_lsst.multi:170 [INFO] [multi] job_id 16887749 proc_lsst.multi INFO: [multi] job_id 16887749 2024-03-07 07:25:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887749 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887749 parsl.executors.status_handling DEBUG: Launched block 1 on executor multi with job ID 16887749 parsl.executors.status_handling INFO: Allocated block ID 2 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-2 2024-03-07 07:25:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-2 2024-03-07 07:25:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=2 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 447 try 0 launched on executor multi with executor id 150 parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/3c33c95c-a77e-4448-b50c-696b79df0161_isr_1124707_54.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/3c33c95c-a77e-4448-b50c-696b79df0161_isr_1124707_54.stderr parsl.dataflow.dflow DEBUG: Task 448 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 448 submitted for App characterizeImage, waiting on task 447 parsl.dataflow.dflow DEBUG: Task 448 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 448 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 449 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 449 submitted for App calibrate, waiting on task 448 parsl.dataflow.dflow DEBUG: Task 449 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 449 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 450 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 450 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 450 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 450 has memoization hash f59c260d3c0225241598825ead89d2f4 parsl.dataflow.memoization INFO: Task 450 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 450 try 0 launched on executor multi with executor id 151 parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/af034429-28fc-4a94-9320-ea1c3d37d6e7_isr_1124710_60.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/af034429-28fc-4a94-9320-ea1c3d37d6e7_isr_1124710_60.stderr parsl.dataflow.dflow DEBUG: Task 451 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 451 submitted for App characterizeImage, waiting on task 450 parsl.dataflow.dflow DEBUG: Task 451 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 451 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 452 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 452 submitted for App calibrate, waiting on task 451 parsl.dataflow.dflow DEBUG: Task 452 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 452 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 453 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 453 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 453 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 453 has memoization hash 1e4da962dc9d465cd37299e3182a3c52 parsl.dataflow.memoization INFO: Task 453 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 453 try 0 launched on executor multi with executor id 152 parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/72b8c9f4-802d-405e-8c45-c652cbe180a6_isr_1124710_12.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/72b8c9f4-802d-405e-8c45-c652cbe180a6_isr_1124710_12.stderr parsl.dataflow.dflow DEBUG: Task 454 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 454 submitted for App characterizeImage, waiting on task 453 parsl.dataflow.dflow DEBUG: Task 454 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 454 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 455 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 455 submitted for App calibrate, waiting on task 454 parsl.dataflow.dflow DEBUG: Task 455 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 455 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 456 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 456 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 456 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 456 has memoization hash 86ede9c59684c015d512af04b5d50ab3 parsl.dataflow.memoization INFO: Task 456 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 456 try 0 launched on executor multi with executor id 153 parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/2d42e3b2-a88b-4fed-bdcd-c4fb0ebf644c_isr_1124828_35.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/2d42e3b2-a88b-4fed-bdcd-c4fb0ebf644c_isr_1124828_35.stderr parsl.dataflow.dflow DEBUG: Task 457 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 457 submitted for App characterizeImage, waiting on task 456 parsl.dataflow.dflow DEBUG: Task 457 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 457 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 458 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 458 submitted for App calibrate, waiting on task 457 parsl.dataflow.dflow DEBUG: Task 458 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 458 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 459 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 459 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 459 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 459 has memoization hash e0dc03bae600f1cbaedec6fc48cc97e3 parsl.dataflow.memoization INFO: Task 459 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 459 try 0 launched on executor multi with executor id 154 parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/6c37fa58-4db2-4ff2-a65b-36c5a747c41a_isr_1124710_52.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/6c37fa58-4db2-4ff2-a65b-36c5a747c41a_isr_1124710_52.stderr parsl.dataflow.dflow DEBUG: Task 460 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 460 submitted for App characterizeImage, waiting on task 459 parsl.dataflow.dflow DEBUG: Task 460 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 460 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 461 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 461 submitted for App calibrate, waiting on task 460 parsl.dataflow.dflow DEBUG: Task 461 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 461 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 462 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 462 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 462 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 462 has memoization hash 2326872e9f9abc010cc103a59693c4a3 parsl.dataflow.memoization INFO: Task 462 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529851e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 462 try 0 launched on executor multi with executor id 155 parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/8134fbd9-ddf8-43ed-98d4-cb4f306a5c49_isr_1124831_46.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/8134fbd9-ddf8-43ed-98d4-cb4f306a5c49_isr_1124831_46.stderr parsl.dataflow.dflow DEBUG: Task 463 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 463 submitted for App characterizeImage, waiting on task 462 parsl.dataflow.dflow DEBUG: Task 463 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 463 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 464 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 464 submitted for App calibrate, waiting on task 463 parsl.dataflow.dflow DEBUG: Task 464 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 464 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 465 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 465 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 465 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 465 has memoization hash 93a740181253dbd75f01031af89e5042 parsl.dataflow.memoization INFO: Task 465 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 465 try 0 launched on executor multi with executor id 156 parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7071c0ff-8b8d-43b6-a3a4-380fc447f612_isr_1124831_56.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7071c0ff-8b8d-43b6-a3a4-380fc447f612_isr_1124831_56.stderr parsl.dataflow.dflow DEBUG: Task 466 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 466 submitted for App characterizeImage, waiting on task 465 parsl.dataflow.dflow DEBUG: Task 466 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 466 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 467 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 467 submitted for App calibrate, waiting on task 466 parsl.dataflow.dflow DEBUG: Task 467 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 467 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 468 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 468 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 468 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 468 has memoization hash 2aa09b8e58040f98c6f1b6ee5874544d parsl.dataflow.memoization INFO: Task 468 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 468 try 0 launched on executor multi with executor id 157 parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/410e1a24-2665-4d29-89bc-67cbd7003a8b_isr_1124831_54.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/410e1a24-2665-4d29-89bc-67cbd7003a8b_isr_1124831_54.stderr parsl.dataflow.dflow DEBUG: Task 469 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 469 submitted for App characterizeImage, waiting on task 468 parsl.dataflow.dflow DEBUG: Task 469 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 469 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 470 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 470 submitted for App calibrate, waiting on task 469 parsl.dataflow.dflow DEBUG: Task 470 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 470 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 471 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 471 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 471 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 471 has memoization hash 6eb7e821f926b9cdaa5e0fb7a860b1c9 parsl.dataflow.memoization INFO: Task 471 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 471 try 0 launched on executor multi with executor id 158 parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/900eb175-97e0-465d-9bce-043b0336b0db_isr_1124710_47.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/900eb175-97e0-465d-9bce-043b0336b0db_isr_1124710_47.stderr parsl.dataflow.dflow DEBUG: Task 472 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 472 submitted for App characterizeImage, waiting on task 471 parsl.dataflow.dflow DEBUG: Task 472 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 472 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 473 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 473 submitted for App calibrate, waiting on task 472 parsl.dataflow.dflow DEBUG: Task 473 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 473 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 474 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 474 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 474 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 474 has memoization hash c8444714c030d45c3a221e3bbf1258f8 parsl.dataflow.memoization INFO: Task 474 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529851c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 474 try 0 launched on executor multi with executor id 159 parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/7fee85d2-77ec-4396-bb8c-5c0f4901cf1f_isr_1124707_11.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/7fee85d2-77ec-4396-bb8c-5c0f4901cf1f_isr_1124707_11.stderr parsl.dataflow.dflow DEBUG: Task 475 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 475 submitted for App characterizeImage, waiting on task 474 parsl.dataflow.dflow DEBUG: Task 475 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 475 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 476 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 476 submitted for App calibrate, waiting on task 475 parsl.dataflow.dflow DEBUG: Task 476 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 476 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 477 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 477 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 477 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 477 has memoization hash 39f24eac014013c0eba3b68a74e4d79f parsl.dataflow.memoization INFO: Task 477 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529851da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 477 try 0 launched on executor multi with executor id 160 parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/683e865c-cbf2-4926-821c-d0fd53d94aa6_isr_1124704_11.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/683e865c-cbf2-4926-821c-d0fd53d94aa6_isr_1124704_11.stderr parsl.dataflow.dflow DEBUG: Task 478 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 478 submitted for App characterizeImage, waiting on task 477 parsl.dataflow.dflow DEBUG: Task 478 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 478 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 479 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 479 submitted for App calibrate, waiting on task 478 parsl.dataflow.dflow DEBUG: Task 479 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 479 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 480 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 480 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 480 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 480 has memoization hash efe23a340771169f9ab2a7bd4f6c0993 parsl.dataflow.memoization INFO: Task 480 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529850360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 480 try 0 launched on executor multi with executor id 161 parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/ad8af73f-e0ac-441f-bf89-d6ca9733cacd_isr_1124831_16.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/ad8af73f-e0ac-441f-bf89-d6ca9733cacd_isr_1124831_16.stderr parsl.dataflow.dflow DEBUG: Task 481 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 481 submitted for App characterizeImage, waiting on task 480 parsl.dataflow.dflow DEBUG: Task 481 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 481 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 482 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 482 submitted for App calibrate, waiting on task 481 parsl.dataflow.dflow DEBUG: Task 482 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 482 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 483 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 483 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 483 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 483 has memoization hash 6de05e5758c75916ff21f5be78600789 parsl.dataflow.memoization INFO: Task 483 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529850680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 483 try 0 launched on executor multi with executor id 162 parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/993d57d2-ff78-4686-9911-401ce7cca79e_isr_1124710_34.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/993d57d2-ff78-4686-9911-401ce7cca79e_isr_1124710_34.stderr parsl.dataflow.dflow DEBUG: Task 484 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 484 submitted for App characterizeImage, waiting on task 483 parsl.dataflow.dflow DEBUG: Task 484 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 484 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 485 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 485 submitted for App calibrate, waiting on task 484 parsl.dataflow.dflow DEBUG: Task 485 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 485 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 486 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 486 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 486 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 486 has memoization hash b1b9622d75965056d21ac86108cca312 parsl.dataflow.memoization INFO: Task 486 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298509a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 486 try 0 launched on executor multi with executor id 163 parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/adf876a1-27de-4f19-93c8-20b6b29e8301_isr_1124707_51.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/adf876a1-27de-4f19-93c8-20b6b29e8301_isr_1124707_51.stderr parsl.dataflow.dflow DEBUG: Task 487 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 487 submitted for App characterizeImage, waiting on task 486 parsl.dataflow.dflow DEBUG: Task 487 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 487 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 488 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 488 submitted for App calibrate, waiting on task 487 parsl.dataflow.dflow DEBUG: Task 488 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 488 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 489 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 489 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 489 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 489 has memoization hash b072cae0cc916837f19a575f0f3362de parsl.dataflow.memoization INFO: Task 489 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529850a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 489 try 0 launched on executor multi with executor id 164 parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/09d560f6-7767-42c8-8d98-887fc49c25b5_isr_1124831_32.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/09d560f6-7767-42c8-8d98-887fc49c25b5_isr_1124831_32.stderr parsl.dataflow.dflow DEBUG: Task 490 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 490 submitted for App characterizeImage, waiting on task 489 parsl.dataflow.dflow DEBUG: Task 490 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 490 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 491 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 491 submitted for App calibrate, waiting on task 490 parsl.dataflow.dflow DEBUG: Task 491 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 491 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 492 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 492 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 492 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 492 has memoization hash 1116e3df0a5dbbbd48bbd0dec4db0a73 parsl.dataflow.memoization INFO: Task 492 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529850f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 492 try 0 launched on executor multi with executor id 165 parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/44353e88-1e3e-498c-a41b-fbff07cd7786_isr_1124828_20.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/44353e88-1e3e-498c-a41b-fbff07cd7786_isr_1124828_20.stderr parsl.dataflow.dflow DEBUG: Task 493 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 493 submitted for App characterizeImage, waiting on task 492 parsl.dataflow.dflow DEBUG: Task 493 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 493 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 494 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 494 submitted for App calibrate, waiting on task 493 parsl.dataflow.dflow DEBUG: Task 494 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 494 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 495 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 495 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 495 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 495 has memoization hash d4e02ba6726739ce7074a7b6d3ad7dd0 parsl.dataflow.memoization INFO: Task 495 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529851b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 495 try 0 launched on executor multi with executor id 166 parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c0bd55ef-057b-458f-8f55-c8b125f26f59_isr_1124707_40.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c0bd55ef-057b-458f-8f55-c8b125f26f59_isr_1124707_40.stderr parsl.dataflow.dflow DEBUG: Task 496 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 496 submitted for App characterizeImage, waiting on task 495 parsl.dataflow.dflow DEBUG: Task 496 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 496 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 497 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 497 submitted for App calibrate, waiting on task 496 parsl.dataflow.dflow DEBUG: Task 497 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 497 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 498 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 498 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 498 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 498 has memoization hash 6bec64eedc925f384608ed93cf1c2985 parsl.dataflow.memoization INFO: Task 498 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529851800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 498 try 0 launched on executor multi with executor id 167 parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/6146528f-560e-418d-8d1e-31b90e334f9c_isr_1124831_13.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/6146528f-560e-418d-8d1e-31b90e334f9c_isr_1124831_13.stderr parsl.dataflow.dflow DEBUG: Task 499 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 499 submitted for App characterizeImage, waiting on task 498 parsl.dataflow.dflow DEBUG: Task 499 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 499 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 500 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 500 submitted for App calibrate, waiting on task 499 parsl.dataflow.dflow DEBUG: Task 500 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 500 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 501 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 501 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 501 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 501 has memoization hash c91f404907ff9cad257444aa291eda34 parsl.dataflow.memoization INFO: Task 501 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298514e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 501 try 0 launched on executor multi with executor id 168 parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/2887118a-7013-432c-a361-6fbd9c613ac3_isr_1124707_53.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/2887118a-7013-432c-a361-6fbd9c613ac3_isr_1124707_53.stderr parsl.dataflow.dflow DEBUG: Task 502 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 502 submitted for App characterizeImage, waiting on task 501 parsl.dataflow.dflow DEBUG: Task 502 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 502 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 503 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 503 submitted for App calibrate, waiting on task 502 parsl.dataflow.dflow DEBUG: Task 503 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 503 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 504 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 504 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 504 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 504 has memoization hash 88ffaabb9024c1f25e86a0a2bc279f77 parsl.dataflow.memoization INFO: Task 504 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298511c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 504 try 0 launched on executor multi with executor id 169 parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/25d89f12-1d48-4bd8-8e8b-6651fbb7cd67_isr_1124831_4.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/25d89f12-1d48-4bd8-8e8b-6651fbb7cd67_isr_1124831_4.stderr parsl.dataflow.dflow DEBUG: Task 505 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 505 submitted for App characterizeImage, waiting on task 504 parsl.dataflow.dflow DEBUG: Task 505 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 505 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 506 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 506 submitted for App calibrate, waiting on task 505 parsl.dataflow.dflow DEBUG: Task 506 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 506 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 507 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 507 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 507 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 507 has memoization hash 49a9fdd1979e49de45fd56c7586edfe1 parsl.dataflow.memoization INFO: Task 507 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298527a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 07:25:41 proc_lsst.multi:170 [INFO] [multi] job_id 16887750 proc_lsst.multi INFO: [multi] job_id 16887750 2024-03-07 07:25:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 07:25:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887750 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887750 parsl.executors.status_handling DEBUG: Launched block 2 on executor multi with job ID 16887750 parsl.executors.status_handling INFO: Allocated block ID 3 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-3 2024-03-07 07:25:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-3 2024-03-07 07:25:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 07:25:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=3 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 507 try 0 launched on executor multi with executor id 170 parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/3858304d-3351-4beb-a153-3b9cd12eacca_isr_1124704_3.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/3858304d-3351-4beb-a153-3b9cd12eacca_isr_1124704_3.stderr parsl.dataflow.dflow DEBUG: Task 508 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 508 submitted for App characterizeImage, waiting on task 507 parsl.dataflow.dflow DEBUG: Task 508 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 508 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 509 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 509 submitted for App calibrate, waiting on task 508 parsl.dataflow.dflow DEBUG: Task 509 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 509 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 510 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 510 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 510 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 510 has memoization hash c7798f95312077a220093f9940937889 parsl.dataflow.memoization INFO: Task 510 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 510 try 0 launched on executor multi with executor id 171 parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7e53e051-f319-4ffd-a4b5-ee49521cc43b_isr_1124831_50.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7e53e051-f319-4ffd-a4b5-ee49521cc43b_isr_1124831_50.stderr parsl.dataflow.dflow DEBUG: Task 511 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 511 submitted for App characterizeImage, waiting on task 510 parsl.dataflow.dflow DEBUG: Task 511 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 511 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 512 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 512 submitted for App calibrate, waiting on task 511 parsl.dataflow.dflow DEBUG: Task 512 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 512 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 513 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 513 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 513 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 513 has memoization hash 91d0db657723b82e4971dd13e88dcc42 parsl.dataflow.memoization INFO: Task 513 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852ca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 513 try 0 launched on executor multi with executor id 172 parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/217cb1bc-ee61-4834-bcf3-8a0ea153d299_isr_1124710_62.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/217cb1bc-ee61-4834-bcf3-8a0ea153d299_isr_1124710_62.stderr parsl.dataflow.dflow DEBUG: Task 514 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 514 submitted for App characterizeImage, waiting on task 513 parsl.dataflow.dflow DEBUG: Task 514 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 514 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 515 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 515 submitted for App calibrate, waiting on task 514 parsl.dataflow.dflow DEBUG: Task 515 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 515 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 516 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 516 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 516 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.memoization DEBUG: Task 516 has memoization hash 4d2e3f24680b18a343af4d80c84c50d7 parsl.dataflow.memoization INFO: Task 516 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852fc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 516 try 0 launched on executor multi with executor id 173 parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/55b72295-7b64-4fca-8e4c-86036301bde2_isr_1124831_24.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/55b72295-7b64-4fca-8e4c-86036301bde2_isr_1124831_24.stderr parsl.dataflow.dflow DEBUG: Task 517 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 517 submitted for App characterizeImage, waiting on task 516 parsl.dataflow.dflow DEBUG: Task 517 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 517 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 518 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 518 submitted for App calibrate, waiting on task 517 parsl.dataflow.dflow DEBUG: Task 518 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 518 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 519 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 519 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 519 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 519 has memoization hash 5ab5d7fd1dd66e81541d4b0b775d5bcd parsl.dataflow.memoization INFO: Task 519 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529853240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 519 try 0 launched on executor multi with executor id 174 parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/1dc6dea3-9d6c-4993-bc75-a8700590f7ac_isr_1124831_49.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/1dc6dea3-9d6c-4993-bc75-a8700590f7ac_isr_1124831_49.stderr parsl.dataflow.dflow DEBUG: Task 520 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 520 submitted for App characterizeImage, waiting on task 519 parsl.dataflow.dflow DEBUG: Task 520 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 520 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 521 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 521 submitted for App calibrate, waiting on task 520 parsl.dataflow.dflow DEBUG: Task 521 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 521 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 522 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 522 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 522 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 522 has memoization hash ffbac31b8e542f7733c942fee7e3a1ad parsl.dataflow.memoization INFO: Task 522 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 522 try 0 launched on executor multi with executor id 175 parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/8cb735c4-db77-4508-bf76-2acbd33d7bf5_isr_1124831_51.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/8cb735c4-db77-4508-bf76-2acbd33d7bf5_isr_1124831_51.stderr parsl.dataflow.dflow DEBUG: Task 523 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 523 submitted for App characterizeImage, waiting on task 522 parsl.dataflow.dflow DEBUG: Task 523 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 523 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 524 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 524 submitted for App calibrate, waiting on task 523 parsl.dataflow.dflow DEBUG: Task 524 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 524 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 525 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 525 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 525 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 525 has memoization hash e14af678da9eaa094549b20b7514959e parsl.dataflow.memoization INFO: Task 525 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 525 try 0 launched on executor multi with executor id 176 parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a10765d2-cd77-4a85-9696-51fcd061300e_isr_1124710_42.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a10765d2-cd77-4a85-9696-51fcd061300e_isr_1124710_42.stderr parsl.dataflow.dflow DEBUG: Task 526 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 526 submitted for App characterizeImage, waiting on task 525 parsl.dataflow.dflow DEBUG: Task 526 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 526 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 527 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 527 submitted for App calibrate, waiting on task 526 parsl.dataflow.dflow DEBUG: Task 527 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 527 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 528 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 528 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 528 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 528 has memoization hash 2e1f6bc9c17ddda07845e31bf01c1651 parsl.dataflow.memoization INFO: Task 528 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 528 try 0 launched on executor multi with executor id 177 parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/133eb39e-8dd0-4f8e-a1fc-8388228a6fee_isr_1124828_50.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/133eb39e-8dd0-4f8e-a1fc-8388228a6fee_isr_1124828_50.stderr parsl.dataflow.dflow DEBUG: Task 529 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 529 submitted for App characterizeImage, waiting on task 528 parsl.dataflow.dflow DEBUG: Task 529 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 529 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 530 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 530 submitted for App calibrate, waiting on task 529 parsl.dataflow.dflow DEBUG: Task 530 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 530 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 531 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 531 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 531 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 531 has memoization hash f22c37b4c4288365b628c708b455d773 parsl.dataflow.memoization INFO: Task 531 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529853ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 531 try 0 launched on executor multi with executor id 178 parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/068840a4-e57c-43a7-9922-d7d1d29ec731_isr_1124710_35.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/068840a4-e57c-43a7-9922-d7d1d29ec731_isr_1124710_35.stderr parsl.dataflow.dflow DEBUG: Task 532 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 532 submitted for App characterizeImage, waiting on task 531 parsl.dataflow.dflow DEBUG: Task 532 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 532 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 533 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 533 submitted for App calibrate, waiting on task 532 parsl.dataflow.dflow DEBUG: Task 533 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 533 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 534 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 534 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 534 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 534 has memoization hash d57cc497b5e9039cc299e8fe8f7a3771 parsl.dataflow.memoization INFO: Task 534 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529853ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 534 try 0 launched on executor multi with executor id 179 parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/1448a12f-91f2-4e44-bafc-e195593a6a46_isr_1124707_12.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/1448a12f-91f2-4e44-bafc-e195593a6a46_isr_1124707_12.stderr parsl.dataflow.dflow DEBUG: Task 535 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 535 submitted for App characterizeImage, waiting on task 534 parsl.dataflow.dflow DEBUG: Task 535 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 535 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 536 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 536 submitted for App calibrate, waiting on task 535 parsl.dataflow.dflow DEBUG: Task 536 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 536 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 537 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 537 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 537 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 537 has memoization hash bb09180fd244159b7faf24a558cbbef5 parsl.dataflow.memoization INFO: Task 537 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 537 try 0 launched on executor multi with executor id 180 parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/7dd647cf-219b-4b8e-b775-c55064c897df_isr_1124707_31.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/7dd647cf-219b-4b8e-b775-c55064c897df_isr_1124707_31.stderr parsl.dataflow.dflow DEBUG: Task 538 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 538 submitted for App characterizeImage, waiting on task 537 parsl.dataflow.dflow DEBUG: Task 538 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 538 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 539 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 539 submitted for App calibrate, waiting on task 538 parsl.dataflow.dflow DEBUG: Task 539 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 539 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 540 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 540 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 540 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 540 has memoization hash 0f1380623dfe389922fe450b25764eb7 parsl.dataflow.memoization INFO: Task 540 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 540 try 0 launched on executor multi with executor id 181 parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/42c6a149-5575-4d55-bb65-06e035a599be_isr_1124704_33.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/42c6a149-5575-4d55-bb65-06e035a599be_isr_1124704_33.stderr parsl.dataflow.dflow DEBUG: Task 541 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 541 submitted for App characterizeImage, waiting on task 540 parsl.dataflow.dflow DEBUG: Task 541 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 541 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 542 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 542 submitted for App calibrate, waiting on task 541 parsl.dataflow.dflow DEBUG: Task 542 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 542 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 543 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 543 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 543 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 543 has memoization hash 8f5100e0c6ac4d27fe549f63104758b0 parsl.dataflow.memoization INFO: Task 543 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 543 try 0 launched on executor multi with executor id 182 parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/b918198a-2942-409a-9e1e-fa0e4821e77e_isr_1124710_29.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/b918198a-2942-409a-9e1e-fa0e4821e77e_isr_1124710_29.stderr parsl.dataflow.dflow DEBUG: Task 544 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 544 submitted for App characterizeImage, waiting on task 543 parsl.dataflow.dflow DEBUG: Task 544 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 544 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 545 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 545 submitted for App calibrate, waiting on task 544 parsl.dataflow.dflow DEBUG: Task 545 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 545 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 546 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 546 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 546 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 546 has memoization hash a91ddec92762f91efe9fb679337de558 parsl.dataflow.memoization INFO: Task 546 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 546 try 0 launched on executor multi with executor id 183 parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1af31943-db04-4be3-b345-e39f7af1fd63_isr_1124710_4.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1af31943-db04-4be3-b345-e39f7af1fd63_isr_1124710_4.stderr parsl.dataflow.dflow DEBUG: Task 547 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 547 submitted for App characterizeImage, waiting on task 546 parsl.dataflow.dflow DEBUG: Task 547 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 547 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 548 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 548 submitted for App calibrate, waiting on task 547 parsl.dataflow.dflow DEBUG: Task 548 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 548 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 549 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 549 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 549 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 549 has memoization hash 12560c3662a6eefdf58763ba855e6dc7 parsl.dataflow.memoization INFO: Task 549 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 549 try 0 launched on executor multi with executor id 184 parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/f2fc72bf-44fb-4558-bd6d-55c042ae7986_isr_1124704_53.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/f2fc72bf-44fb-4558-bd6d-55c042ae7986_isr_1124704_53.stderr parsl.dataflow.dflow DEBUG: Task 550 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 550 submitted for App characterizeImage, waiting on task 549 parsl.dataflow.dflow DEBUG: Task 550 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 550 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 551 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 551 submitted for App calibrate, waiting on task 550 parsl.dataflow.dflow DEBUG: Task 551 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 551 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 552 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 552 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 552 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 552 has memoization hash 645ac3365da7757f3db5c6e434148025 parsl.dataflow.memoization INFO: Task 552 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 552 try 0 launched on executor multi with executor id 185 parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/221ade2c-9a3e-4325-bc54-31b66bc2bb6d_isr_1124831_8.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/221ade2c-9a3e-4325-bc54-31b66bc2bb6d_isr_1124831_8.stderr parsl.dataflow.dflow DEBUG: Task 553 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 553 submitted for App characterizeImage, waiting on task 552 parsl.dataflow.dflow DEBUG: Task 553 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 553 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 554 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 554 submitted for App calibrate, waiting on task 553 parsl.dataflow.dflow DEBUG: Task 554 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 554 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 555 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 555 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 555 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 555 has memoization hash 5ae13842035feeaeecc6c0607e2c6978 parsl.dataflow.memoization INFO: Task 555 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969d120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 555 try 0 launched on executor multi with executor id 186 parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/4e78c99f-11fe-497a-86dd-a12acfc0d8bf_isr_1124704_17.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/4e78c99f-11fe-497a-86dd-a12acfc0d8bf_isr_1124704_17.stderr parsl.dataflow.dflow DEBUG: Task 556 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 556 submitted for App characterizeImage, waiting on task 555 parsl.dataflow.dflow DEBUG: Task 556 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 556 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 557 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 557 submitted for App calibrate, waiting on task 556 parsl.dataflow.dflow DEBUG: Task 557 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 557 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 558 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 558 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 558 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 558 has memoization hash e8aca9db8cfc5c82fb556e93c53b0ab8 parsl.dataflow.memoization INFO: Task 558 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 558 try 0 launched on executor multi with executor id 187 parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/3460dbcc-a284-44be-b6bf-ddaf3866d6b2_isr_1124831_9.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/3460dbcc-a284-44be-b6bf-ddaf3866d6b2_isr_1124831_9.stderr parsl.dataflow.dflow DEBUG: Task 559 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 559 submitted for App characterizeImage, waiting on task 558 parsl.dataflow.dflow DEBUG: Task 559 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 559 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 560 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 560 submitted for App calibrate, waiting on task 559 parsl.dataflow.dflow DEBUG: Task 560 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 560 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 561 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 561 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 561 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 561 has memoization hash d764a3613366b2042a47cf17150cb727 parsl.dataflow.memoization INFO: Task 561 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 561 try 0 launched on executor multi with executor id 188 parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/44546abd-d3f5-4ec1-8e40-331c5b87be9a_isr_1124707_26.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/44546abd-d3f5-4ec1-8e40-331c5b87be9a_isr_1124707_26.stderr parsl.dataflow.dflow DEBUG: Task 562 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 562 submitted for App characterizeImage, waiting on task 561 parsl.dataflow.dflow DEBUG: Task 562 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 562 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 563 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 563 submitted for App calibrate, waiting on task 562 parsl.dataflow.dflow DEBUG: Task 563 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 563 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 564 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 564 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 564 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 564 has memoization hash 7bb3981b8651653807e20010c90c7dc8 parsl.dataflow.memoization INFO: Task 564 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 564 try 0 launched on executor multi with executor id 189 parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/0e85c742-38c5-484a-a504-af0faae07a60_isr_1124828_33.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/0e85c742-38c5-484a-a504-af0faae07a60_isr_1124828_33.stderr parsl.dataflow.dflow DEBUG: Task 565 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 565 submitted for App characterizeImage, waiting on task 564 parsl.dataflow.dflow DEBUG: Task 565 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 565 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 566 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 566 submitted for App calibrate, waiting on task 565 parsl.dataflow.dflow DEBUG: Task 566 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 566 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 567 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 567 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 567 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 567 has memoization hash 278810c18211f830738811af24bbaf19 parsl.dataflow.memoization INFO: Task 567 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 567 try 0 launched on executor multi with executor id 190 parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/1c99e26d-ddad-4b8e-b4c3-7a411fa1483d_isr_1124831_37.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/1c99e26d-ddad-4b8e-b4c3-7a411fa1483d_isr_1124831_37.stderr parsl.dataflow.dflow DEBUG: Task 568 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 568 submitted for App characterizeImage, waiting on task 567 parsl.dataflow.dflow DEBUG: Task 568 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 568 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 569 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 569 submitted for App calibrate, waiting on task 568 parsl.dataflow.dflow DEBUG: Task 569 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 569 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 570 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 570 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 570 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 570 has memoization hash ad45d002290cd4080b5c8318a67a9be7 parsl.dataflow.memoization INFO: Task 570 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 570 try 0 launched on executor multi with executor id 191 parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/830a774e-fe45-47b2-bea5-165fee4a7f06_isr_1124831_22.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/830a774e-fe45-47b2-bea5-165fee4a7f06_isr_1124831_22.stderr parsl.dataflow.dflow DEBUG: Task 571 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 571 submitted for App characterizeImage, waiting on task 570 parsl.dataflow.dflow DEBUG: Task 571 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 571 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 572 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 572 submitted for App calibrate, waiting on task 571 parsl.dataflow.dflow DEBUG: Task 572 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 572 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 573 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 573 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 573 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 573 has memoization hash 4c87ec30508469c74f96a5a848ece585 parsl.dataflow.memoization INFO: Task 573 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 573 try 0 launched on executor multi with executor id 192 parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/c23fdd69-5589-4cfe-893e-e839d082b4a2_isr_1124704_42.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/c23fdd69-5589-4cfe-893e-e839d082b4a2_isr_1124704_42.stderr parsl.dataflow.dflow DEBUG: Task 574 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 574 submitted for App characterizeImage, waiting on task 573 parsl.dataflow.dflow DEBUG: Task 574 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 574 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 575 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 575 submitted for App calibrate, waiting on task 574 parsl.dataflow.dflow DEBUG: Task 575 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 575 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 576 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 576 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 576 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 576 has memoization hash 6081c5e8f580ffab56231aff98efb278 parsl.dataflow.memoization INFO: Task 576 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 576 try 0 launched on executor multi with executor id 193 parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/feb34aab-6bf5-48b7-8876-2d45ef173ecd_isr_1124831_55.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/feb34aab-6bf5-48b7-8876-2d45ef173ecd_isr_1124831_55.stderr parsl.dataflow.dflow DEBUG: Task 577 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 577 submitted for App characterizeImage, waiting on task 576 parsl.dataflow.dflow DEBUG: Task 577 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 577 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 578 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 578 submitted for App calibrate, waiting on task 577 parsl.dataflow.dflow DEBUG: Task 578 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 578 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 579 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 579 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 579 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 579 has memoization hash 4ca473b9bcbccf27d2e00463958fc3ff parsl.dataflow.memoization INFO: Task 579 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 07:25:41 proc_lsst.multi:170 [INFO] [multi] job_id 16887751 proc_lsst.multi INFO: [multi] job_id 16887751 2024-03-07 07:25:41 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 07:25:41 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:41 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887751 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887751 parsl.executors.status_handling DEBUG: Launched block 3 on executor multi with job ID 16887751 parsl.executors.status_handling INFO: Allocated block ID 4 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-4 2024-03-07 07:25:41 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-4 2024-03-07 07:25:41 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:41 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:41 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 07:25:41 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:41 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=4 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 579 try 0 launched on executor multi with executor id 194 parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/29f42147-eeb4-47cc-ac6c-d2afc5f49659_isr_1124707_32.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/29f42147-eeb4-47cc-ac6c-d2afc5f49659_isr_1124707_32.stderr parsl.dataflow.dflow DEBUG: Task 580 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 580 submitted for App characterizeImage, waiting on task 579 parsl.dataflow.dflow DEBUG: Task 580 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 580 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 581 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 581 submitted for App calibrate, waiting on task 580 parsl.dataflow.dflow DEBUG: Task 581 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 581 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 582 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 582 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 582 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 582 has memoization hash 213199a382eca8cf59c2458760831f90 parsl.dataflow.memoization INFO: Task 582 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969f380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 582 try 0 launched on executor multi with executor id 195 parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/78c7e4db-12f0-49b9-9f7a-2f5c0f51f6df_isr_1124828_54.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/78c7e4db-12f0-49b9-9f7a-2f5c0f51f6df_isr_1124828_54.stderr parsl.dataflow.dflow DEBUG: Task 583 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 583 submitted for App characterizeImage, waiting on task 582 parsl.dataflow.dflow DEBUG: Task 583 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 583 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 584 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 584 submitted for App calibrate, waiting on task 583 parsl.dataflow.dflow DEBUG: Task 584 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 584 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 585 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 585 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 585 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 585 has memoization hash 7167a05ef1c3a9c75d78282b8ee72c57 parsl.dataflow.memoization INFO: Task 585 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 585 try 0 launched on executor multi with executor id 196 parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c5c78fcd-622a-4f63-a3a1-e85ec445c980_isr_1124707_21.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c5c78fcd-622a-4f63-a3a1-e85ec445c980_isr_1124707_21.stderr parsl.dataflow.dflow DEBUG: Task 586 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 586 submitted for App characterizeImage, waiting on task 585 parsl.dataflow.dflow DEBUG: Task 586 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 586 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 587 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 587 submitted for App calibrate, waiting on task 586 parsl.dataflow.dflow DEBUG: Task 587 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 587 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 588 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 588 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 588 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 588 has memoization hash 596194199623e189c178d42bcf5705ad parsl.dataflow.memoization INFO: Task 588 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ead40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 588 try 0 launched on executor multi with executor id 197 parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/19b95d54-815a-4717-bbb8-7ddbbfeff40f_isr_1124704_35.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/19b95d54-815a-4717-bbb8-7ddbbfeff40f_isr_1124704_35.stderr parsl.dataflow.dflow DEBUG: Task 589 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 589 submitted for App characterizeImage, waiting on task 588 parsl.dataflow.dflow DEBUG: Task 589 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 589 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 590 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 590 submitted for App calibrate, waiting on task 589 parsl.dataflow.dflow DEBUG: Task 590 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 590 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 591 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 591 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 591 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 591 has memoization hash 0a9b11b217ab570486f8f75f86c71234 parsl.dataflow.memoization INFO: Task 591 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 591 try 0 launched on executor multi with executor id 198 parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/9e07c330-dc1e-4f28-8b5f-36a4e2702b6e_isr_1124707_2.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/9e07c330-dc1e-4f28-8b5f-36a4e2702b6e_isr_1124707_2.stderr parsl.dataflow.dflow DEBUG: Task 592 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 592 submitted for App characterizeImage, waiting on task 591 parsl.dataflow.dflow DEBUG: Task 592 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 592 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 593 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 593 submitted for App calibrate, waiting on task 592 parsl.dataflow.dflow DEBUG: Task 593 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 593 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 594 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 594 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 594 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 594 has memoization hash dc2294d5b6e30dc1ec4ddfe1d29eba23 parsl.dataflow.memoization INFO: Task 594 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 594 try 0 launched on executor multi with executor id 199 parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/f623c370-fd38-4159-953a-56c2d9dcd1d7_isr_1124707_58.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/f623c370-fd38-4159-953a-56c2d9dcd1d7_isr_1124707_58.stderr parsl.dataflow.dflow DEBUG: Task 595 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 595 submitted for App characterizeImage, waiting on task 594 parsl.dataflow.dflow DEBUG: Task 595 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 595 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 596 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 596 submitted for App calibrate, waiting on task 595 parsl.dataflow.dflow DEBUG: Task 596 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 596 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 597 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 597 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 597 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 597 has memoization hash bd82a5eabe55a5f59a1535c8ef52ec61 parsl.dataflow.memoization INFO: Task 597 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ea980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 597 try 0 launched on executor multi with executor id 200 parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a9e8ea36-75be-4dc7-a85d-36f264bfae8a_isr_1124704_1.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a9e8ea36-75be-4dc7-a85d-36f264bfae8a_isr_1124704_1.stderr parsl.dataflow.dflow DEBUG: Task 598 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 598 submitted for App characterizeImage, waiting on task 597 parsl.dataflow.dflow DEBUG: Task 598 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 598 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 599 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 599 submitted for App calibrate, waiting on task 598 parsl.dataflow.dflow DEBUG: Task 599 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 599 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 600 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 600 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 600 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 600 has memoization hash 5e13886f1b0125790181bf7d4375652e parsl.dataflow.memoization INFO: Task 600 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 600 try 0 launched on executor multi with executor id 201 parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/db0cd417-1bfd-4ccf-a80c-b9cb861fce3e_isr_1124828_10.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/db0cd417-1bfd-4ccf-a80c-b9cb861fce3e_isr_1124828_10.stderr parsl.dataflow.dflow DEBUG: Task 601 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 601 submitted for App characterizeImage, waiting on task 600 parsl.dataflow.dflow DEBUG: Task 601 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 601 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 602 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 602 submitted for App calibrate, waiting on task 601 parsl.dataflow.dflow DEBUG: Task 602 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 602 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 603 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 603 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 603 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 603 has memoization hash b0e7b0abfdb66a513aebe39d58bf391e parsl.dataflow.memoization INFO: Task 603 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 603 try 0 launched on executor multi with executor id 202 parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/351d1f22-2310-42c7-9863-c0419fc0c6a8_isr_1124710_54.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/351d1f22-2310-42c7-9863-c0419fc0c6a8_isr_1124710_54.stderr parsl.dataflow.dflow DEBUG: Task 604 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 604 submitted for App characterizeImage, waiting on task 603 parsl.dataflow.dflow DEBUG: Task 604 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 604 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 605 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 605 submitted for App calibrate, waiting on task 604 parsl.dataflow.dflow DEBUG: Task 605 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 605 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 606 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 606 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 606 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 606 has memoization hash 422eb27efd09a255e8874f0b5457e010 parsl.dataflow.memoization INFO: Task 606 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 606 try 0 launched on executor multi with executor id 203 parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/f412f37b-7b9f-4217-a5d8-a278e44853fc_isr_1124828_1.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/f412f37b-7b9f-4217-a5d8-a278e44853fc_isr_1124828_1.stderr parsl.dataflow.dflow DEBUG: Task 607 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 607 submitted for App characterizeImage, waiting on task 606 parsl.dataflow.dflow DEBUG: Task 607 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 607 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 608 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 608 submitted for App calibrate, waiting on task 607 parsl.dataflow.dflow DEBUG: Task 608 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 608 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 609 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 609 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 609 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 609 has memoization hash e7822b09914827cce77a0f3f9ca90425 parsl.dataflow.memoization INFO: Task 609 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 609 try 0 launched on executor multi with executor id 204 parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/75dc2007-9b7e-454c-a953-23dbfa54b278_isr_1124707_18.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/75dc2007-9b7e-454c-a953-23dbfa54b278_isr_1124707_18.stderr parsl.dataflow.dflow DEBUG: Task 610 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 610 submitted for App characterizeImage, waiting on task 609 parsl.dataflow.dflow DEBUG: Task 610 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 610 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 611 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 611 submitted for App calibrate, waiting on task 610 parsl.dataflow.dflow DEBUG: Task 611 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 611 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 612 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 612 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 612 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 612 has memoization hash 819f92182f723f902337444d337d4eee parsl.dataflow.memoization INFO: Task 612 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 612 try 0 launched on executor multi with executor id 205 parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/dc9a070c-4197-4937-9506-d82a1f385b08_isr_1124710_24.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/dc9a070c-4197-4937-9506-d82a1f385b08_isr_1124710_24.stderr parsl.dataflow.dflow DEBUG: Task 613 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 613 submitted for App characterizeImage, waiting on task 612 parsl.dataflow.dflow DEBUG: Task 613 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 613 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 614 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 614 submitted for App calibrate, waiting on task 613 parsl.dataflow.dflow DEBUG: Task 614 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 614 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 615 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 615 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 615 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 615 has memoization hash 3c161a01aafdf7f30fb6ed3b8d49f5c0 parsl.dataflow.memoization INFO: Task 615 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 615 try 0 launched on executor multi with executor id 206 parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/3806cec8-ca93-4d5f-ba91-8d964c5e31f1_isr_1124710_58.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/3806cec8-ca93-4d5f-ba91-8d964c5e31f1_isr_1124710_58.stderr parsl.dataflow.dflow DEBUG: Task 616 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 616 submitted for App characterizeImage, waiting on task 615 parsl.dataflow.dflow DEBUG: Task 616 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 616 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 617 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 617 submitted for App calibrate, waiting on task 616 parsl.dataflow.dflow DEBUG: Task 617 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 617 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 618 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 618 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 618 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 618 has memoization hash b993d5ed18b3d11215e7b39f734df197 parsl.dataflow.memoization INFO: Task 618 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 618 try 0 launched on executor multi with executor id 207 parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bc8f14dd-de00-4ee3-947a-14e1f9fc6db2_isr_1124828_55.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bc8f14dd-de00-4ee3-947a-14e1f9fc6db2_isr_1124828_55.stderr parsl.dataflow.dflow DEBUG: Task 619 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 619 submitted for App characterizeImage, waiting on task 618 parsl.dataflow.dflow DEBUG: Task 619 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 619 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 620 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 620 submitted for App calibrate, waiting on task 619 parsl.dataflow.dflow DEBUG: Task 620 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 620 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 621 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 621 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 621 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 621 has memoization hash 7eb5cc690ccd97e36b5de8b0659d66d1 parsl.dataflow.memoization INFO: Task 621 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 621 try 0 launched on executor multi with executor id 208 parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/263b5dfa-631c-4d2e-a478-d939197a9b83_isr_1124704_29.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/263b5dfa-631c-4d2e-a478-d939197a9b83_isr_1124704_29.stderr parsl.dataflow.dflow DEBUG: Task 622 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 622 submitted for App characterizeImage, waiting on task 621 parsl.dataflow.dflow DEBUG: Task 622 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 622 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 623 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 623 submitted for App calibrate, waiting on task 622 parsl.dataflow.dflow DEBUG: Task 623 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 623 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 624 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 624 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 624 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 624 has memoization hash 0e2ed1fed239e6f969176d32a1911788 parsl.dataflow.memoization INFO: Task 624 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ea700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 624 try 0 launched on executor multi with executor id 209 parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/c8c83bc3-706b-4912-a018-0992989d7c46_isr_1124710_45.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/c8c83bc3-706b-4912-a018-0992989d7c46_isr_1124710_45.stderr parsl.dataflow.dflow DEBUG: Task 625 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 625 submitted for App characterizeImage, waiting on task 624 parsl.dataflow.dflow DEBUG: Task 625 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 625 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 626 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 626 submitted for App calibrate, waiting on task 625 parsl.dataflow.dflow DEBUG: Task 626 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 626 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 627 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 627 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 627 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 627 has memoization hash 24992708581f4b057172751e01719913 parsl.dataflow.memoization INFO: Task 627 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ea3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 627 try 0 launched on executor multi with executor id 210 parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/4504eb1c-f18d-4446-822f-7abc20a8a2c8_isr_1124831_35.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/4504eb1c-f18d-4446-822f-7abc20a8a2c8_isr_1124831_35.stderr parsl.dataflow.dflow DEBUG: Task 628 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 628 submitted for App characterizeImage, waiting on task 627 parsl.dataflow.dflow DEBUG: Task 628 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 628 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 629 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 629 submitted for App calibrate, waiting on task 628 parsl.dataflow.dflow DEBUG: Task 629 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 629 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 630 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 630 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 630 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 630 has memoization hash 09b8668041ceef1019ba64f074527ee2 parsl.dataflow.memoization INFO: Task 630 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ea0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 630 try 0 launched on executor multi with executor id 211 parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/dd8a025d-27de-43c3-af80-bc0e389106c2_isr_1124828_14.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/dd8a025d-27de-43c3-af80-bc0e389106c2_isr_1124828_14.stderr parsl.dataflow.dflow DEBUG: Task 631 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 631 submitted for App characterizeImage, waiting on task 630 parsl.dataflow.dflow DEBUG: Task 631 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 631 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 632 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 632 submitted for App calibrate, waiting on task 631 parsl.dataflow.dflow DEBUG: Task 632 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 632 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 633 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 633 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 633 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 633 has memoization hash 370c33e9e537851e7310dd13a8200ba6 parsl.dataflow.memoization INFO: Task 633 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 633 try 0 launched on executor multi with executor id 212 parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/6c654ce8-c47f-4792-bf0d-27b1aeccd86f_isr_1124710_21.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/6c654ce8-c47f-4792-bf0d-27b1aeccd86f_isr_1124710_21.stderr parsl.dataflow.dflow DEBUG: Task 634 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 634 submitted for App characterizeImage, waiting on task 633 parsl.dataflow.dflow DEBUG: Task 634 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 634 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 635 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 635 submitted for App calibrate, waiting on task 634 parsl.dataflow.dflow DEBUG: Task 635 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 635 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 636 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 636 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 636 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 636 has memoization hash 53bb6b26757cccf86d5b9dbedc4938a2 parsl.dataflow.memoization INFO: Task 636 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e99e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 636 try 0 launched on executor multi with executor id 213 parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/d2f9dc24-7e05-4169-a108-9a02f25b69a2_isr_1124707_19.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/d2f9dc24-7e05-4169-a108-9a02f25b69a2_isr_1124707_19.stderr parsl.dataflow.dflow DEBUG: Task 637 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 637 submitted for App characterizeImage, waiting on task 636 parsl.dataflow.dflow DEBUG: Task 637 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 637 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 638 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 638 submitted for App calibrate, waiting on task 637 parsl.dataflow.dflow DEBUG: Task 638 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 638 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 639 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 639 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 639 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 639 has memoization hash 119585d52bf305d4b0ed9b7f9145e2e6 parsl.dataflow.memoization INFO: Task 639 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eb1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 639 try 0 launched on executor multi with executor id 214 parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5c572b92-f773-4afc-9992-8061e5234d30_isr_1124707_56.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5c572b92-f773-4afc-9992-8061e5234d30_isr_1124707_56.stderr parsl.dataflow.dflow DEBUG: Task 640 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 640 submitted for App characterizeImage, waiting on task 639 parsl.dataflow.dflow DEBUG: Task 640 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 640 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 641 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 641 submitted for App calibrate, waiting on task 640 parsl.dataflow.dflow DEBUG: Task 641 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 641 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 642 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 642 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 642 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 642 has memoization hash d62557e38ee0c7586b7fefe26ae79f80 parsl.dataflow.memoization INFO: Task 642 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eb4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 642 try 0 launched on executor multi with executor id 215 parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/931477e4-ab28-4028-a86f-02b5a22073ca_isr_1124707_6.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/931477e4-ab28-4028-a86f-02b5a22073ca_isr_1124707_6.stderr parsl.dataflow.dflow DEBUG: Task 643 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 643 submitted for App characterizeImage, waiting on task 642 parsl.dataflow.dflow DEBUG: Task 643 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 643 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 644 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 644 submitted for App calibrate, waiting on task 643 parsl.dataflow.dflow DEBUG: Task 644 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 644 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 645 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 645 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 645 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 645 has memoization hash dfe92bcc7457f7d5ed808d7eac14e8bd parsl.dataflow.memoization INFO: Task 645 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 645 try 0 launched on executor multi with executor id 216 parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1afbedac-bf9f-4918-a17f-e47fc3ef846c_isr_1124710_1.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1afbedac-bf9f-4918-a17f-e47fc3ef846c_isr_1124710_1.stderr parsl.dataflow.dflow DEBUG: Task 646 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 646 submitted for App characterizeImage, waiting on task 645 parsl.dataflow.dflow DEBUG: Task 646 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 646 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 647 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 647 submitted for App calibrate, waiting on task 646 parsl.dataflow.dflow DEBUG: Task 647 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 647 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 648 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 648 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 648 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 648 has memoization hash 9720b36f2ce7c3b316e71ce07098a73e parsl.dataflow.memoization INFO: Task 648 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ebb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 648 try 0 launched on executor multi with executor id 217 parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/c5e1ed19-060b-4e36-9f2a-a3efd3250b5e_isr_1124831_7.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/c5e1ed19-060b-4e36-9f2a-a3efd3250b5e_isr_1124831_7.stderr parsl.dataflow.dflow DEBUG: Task 649 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 649 submitted for App characterizeImage, waiting on task 648 parsl.dataflow.dflow DEBUG: Task 649 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 649 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 650 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 650 submitted for App calibrate, waiting on task 649 parsl.dataflow.dflow DEBUG: Task 650 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 650 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 651 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 651 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 651 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 651 has memoization hash 78044617628f5cca4ef97ed20fa5f464 parsl.dataflow.memoization INFO: Task 651 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ebe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 651 try 0 launched on executor multi with executor id 218 parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0e24fc76-6e41-447c-8568-019ba9745961_isr_1124707_55.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0e24fc76-6e41-447c-8568-019ba9745961_isr_1124707_55.stderr parsl.dataflow.dflow DEBUG: Task 652 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 652 submitted for App characterizeImage, waiting on task 651 parsl.dataflow.dflow DEBUG: Task 652 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 652 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 653 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 653 submitted for App calibrate, waiting on task 652 parsl.dataflow.dflow DEBUG: Task 653 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 653 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 654 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 654 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 654 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 654 has memoization hash c5a17f12b74611c871aca7a994c2eae8 parsl.dataflow.memoization INFO: Task 654 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529738180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 07:25:42 proc_lsst.multi:170 [INFO] [multi] job_id 16887752 proc_lsst.multi INFO: [multi] job_id 16887752 2024-03-07 07:25:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 07:25:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887752 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887752 parsl.executors.status_handling DEBUG: Launched block 4 on executor multi with job ID 16887752 parsl.executors.status_handling INFO: Allocated block ID 5 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-5 2024-03-07 07:25:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-5 2024-03-07 07:25:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 07:25:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=5 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 654 try 0 launched on executor multi with executor id 219 parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f8ae11b2-d715-4bea-92de-09a802de7db6_isr_1124710_19.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f8ae11b2-d715-4bea-92de-09a802de7db6_isr_1124710_19.stderr parsl.dataflow.dflow DEBUG: Task 655 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 655 submitted for App characterizeImage, waiting on task 654 parsl.dataflow.dflow DEBUG: Task 655 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 655 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 656 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 656 submitted for App calibrate, waiting on task 655 parsl.dataflow.dflow DEBUG: Task 656 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 656 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 657 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 657 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 657 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 657 has memoization hash b266794165b0b36d5414d6355b208d39 parsl.dataflow.memoization INFO: Task 657 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529738220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 657 try 0 launched on executor multi with executor id 220 parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a788dc17-c313-4e92-91e7-55aea9bc489b_isr_1124704_30.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a788dc17-c313-4e92-91e7-55aea9bc489b_isr_1124704_30.stderr parsl.dataflow.dflow DEBUG: Task 658 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 658 submitted for App characterizeImage, waiting on task 657 parsl.dataflow.dflow DEBUG: Task 658 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 658 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 659 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 659 submitted for App calibrate, waiting on task 658 parsl.dataflow.dflow DEBUG: Task 659 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 659 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 660 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 660 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 660 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 660 has memoization hash 8213ce06368e93d70fec11c8ffe6e8fc parsl.dataflow.memoization INFO: Task 660 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297385e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 660 try 0 launched on executor multi with executor id 221 parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/15dd493b-3d75-404f-acfa-f49a422efadc_isr_1124710_49.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/15dd493b-3d75-404f-acfa-f49a422efadc_isr_1124710_49.stderr parsl.dataflow.dflow DEBUG: Task 661 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 661 submitted for App characterizeImage, waiting on task 660 parsl.dataflow.dflow DEBUG: Task 661 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 661 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 662 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 662 submitted for App calibrate, waiting on task 661 parsl.dataflow.dflow DEBUG: Task 662 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 662 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 663 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 663 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 663 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 663 has memoization hash 2e2167023b15e8108a0375ac7f265534 parsl.dataflow.memoization INFO: Task 663 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529738900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 663 try 0 launched on executor multi with executor id 222 parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/055cea0c-2a55-45d3-bcfc-76bb629986ca_isr_1124710_39.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/055cea0c-2a55-45d3-bcfc-76bb629986ca_isr_1124710_39.stderr parsl.dataflow.dflow DEBUG: Task 664 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 664 submitted for App characterizeImage, waiting on task 663 parsl.dataflow.dflow DEBUG: Task 664 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 664 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 665 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 665 submitted for App calibrate, waiting on task 664 parsl.dataflow.dflow DEBUG: Task 665 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 665 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 666 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 666 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 666 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 666 has memoization hash 6d7bcf65bb59e97f016972377a07809a parsl.dataflow.memoization INFO: Task 666 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 666 try 0 launched on executor multi with executor id 223 parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/7d51aa65-0980-4a66-90ad-9bf2fb2eca19_isr_1124704_62.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/7d51aa65-0980-4a66-90ad-9bf2fb2eca19_isr_1124704_62.stderr parsl.dataflow.dflow DEBUG: Task 667 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 667 submitted for App characterizeImage, waiting on task 666 parsl.dataflow.dflow DEBUG: Task 667 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 667 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 668 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 668 submitted for App calibrate, waiting on task 667 parsl.dataflow.dflow DEBUG: Task 668 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 668 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 669 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 669 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 669 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 669 has memoization hash 5bf537b5b6d098f6e20a68c4f80be80a parsl.dataflow.memoization INFO: Task 669 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529738e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 669 try 0 launched on executor multi with executor id 224 parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/f1a1a4ca-4c3d-4adf-9038-cadcca3e1637_isr_1124828_47.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/f1a1a4ca-4c3d-4adf-9038-cadcca3e1637_isr_1124828_47.stderr parsl.dataflow.dflow DEBUG: Task 670 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 670 submitted for App characterizeImage, waiting on task 669 parsl.dataflow.dflow DEBUG: Task 670 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 670 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 671 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 671 submitted for App calibrate, waiting on task 670 parsl.dataflow.dflow DEBUG: Task 671 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 671 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 672 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 672 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 672 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 672 has memoization hash c1a35ce72024706e55185640c509ca29 parsl.dataflow.memoization INFO: Task 672 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529739080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 672 try 0 launched on executor multi with executor id 225 parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/2b3ca996-a46d-4047-bec0-8a615537f328_isr_1124704_55.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/2b3ca996-a46d-4047-bec0-8a615537f328_isr_1124704_55.stderr parsl.dataflow.dflow DEBUG: Task 673 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 673 submitted for App characterizeImage, waiting on task 672 parsl.dataflow.dflow DEBUG: Task 673 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 673 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 674 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 674 submitted for App calibrate, waiting on task 673 parsl.dataflow.dflow DEBUG: Task 674 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 674 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 675 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 675 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 675 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 675 has memoization hash 2e89d37be8529d1abae5a29d949d8a0e parsl.dataflow.memoization INFO: Task 675 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297393a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 675 try 0 launched on executor multi with executor id 226 parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/b516c265-2fb4-4ba4-8e7f-21d9d142b022_isr_1124831_10.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/b516c265-2fb4-4ba4-8e7f-21d9d142b022_isr_1124831_10.stderr parsl.dataflow.dflow DEBUG: Task 676 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 676 submitted for App characterizeImage, waiting on task 675 parsl.dataflow.dflow DEBUG: Task 676 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 676 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 677 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 677 submitted for App calibrate, waiting on task 676 parsl.dataflow.dflow DEBUG: Task 677 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 677 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 678 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 678 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 678 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 678 has memoization hash 427cc7b3a04ec7f88b59d09a4dc7517c parsl.dataflow.memoization INFO: Task 678 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297394e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 678 try 0 launched on executor multi with executor id 227 parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/0b9172f4-99b7-4c4d-adf0-9d9c26a48318_isr_1124710_3.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/0b9172f4-99b7-4c4d-adf0-9d9c26a48318_isr_1124710_3.stderr parsl.dataflow.dflow DEBUG: Task 679 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 679 submitted for App characterizeImage, waiting on task 678 parsl.dataflow.dflow DEBUG: Task 679 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 679 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 680 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 680 submitted for App calibrate, waiting on task 679 parsl.dataflow.dflow DEBUG: Task 680 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 680 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 681 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 681 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 681 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 681 has memoization hash c705a5609630110653986a9341ec14fa parsl.dataflow.memoization INFO: Task 681 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529739800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 681 try 0 launched on executor multi with executor id 228 parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/3c349fe0-410a-476c-bd50-265e41e27a1b_isr_1124710_11.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/3c349fe0-410a-476c-bd50-265e41e27a1b_isr_1124710_11.stderr parsl.dataflow.dflow DEBUG: Task 682 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 682 submitted for App characterizeImage, waiting on task 681 parsl.dataflow.dflow DEBUG: Task 682 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 682 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 683 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 683 submitted for App calibrate, waiting on task 682 parsl.dataflow.dflow DEBUG: Task 683 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 683 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 684 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 684 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 684 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 684 has memoization hash 6f3d87e017ff764cb604570c9e070093 parsl.dataflow.memoization INFO: Task 684 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529739c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 684 try 0 launched on executor multi with executor id 229 parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/d0c753d0-a635-4579-8f87-866372527993_isr_1124707_4.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/d0c753d0-a635-4579-8f87-866372527993_isr_1124707_4.stderr parsl.dataflow.dflow DEBUG: Task 685 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 685 submitted for App characterizeImage, waiting on task 684 parsl.dataflow.dflow DEBUG: Task 685 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 685 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 686 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 686 submitted for App calibrate, waiting on task 685 parsl.dataflow.dflow DEBUG: Task 686 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 686 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 687 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 687 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 687 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 687 has memoization hash d1f4f3f1e67d23878212d612b7cb97c2 parsl.dataflow.memoization INFO: Task 687 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529739f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 687 try 0 launched on executor multi with executor id 230 parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e052651e-2c4e-4c43-ab49-2f6d84e55b5e_isr_1124704_21.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e052651e-2c4e-4c43-ab49-2f6d84e55b5e_isr_1124704_21.stderr parsl.dataflow.dflow DEBUG: Task 688 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 688 submitted for App characterizeImage, waiting on task 687 parsl.dataflow.dflow DEBUG: Task 688 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 688 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 689 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 689 submitted for App calibrate, waiting on task 688 parsl.dataflow.dflow DEBUG: Task 689 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 689 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 690 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 690 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 690 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 690 has memoization hash 3e3c7a7d02f236e9778a3cc6b72ee298 parsl.dataflow.memoization INFO: Task 690 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973a200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 690 try 0 launched on executor multi with executor id 231 parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/c9895ecc-9fa4-4145-8d16-53c5fec4c357_isr_1124828_53.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/c9895ecc-9fa4-4145-8d16-53c5fec4c357_isr_1124828_53.stderr parsl.dataflow.dflow DEBUG: Task 691 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 691 submitted for App characterizeImage, waiting on task 690 parsl.dataflow.dflow DEBUG: Task 691 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 691 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 692 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 692 submitted for App calibrate, waiting on task 691 parsl.dataflow.dflow DEBUG: Task 692 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 692 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 693 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 693 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 693 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 693 has memoization hash e17c2bfe5d36e21c4fdcfbe5e3f7c5ee parsl.dataflow.memoization INFO: Task 693 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 693 try 0 launched on executor multi with executor id 232 parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/e382886f-7c7a-4045-aab3-ab3aa40c6b51_isr_1124828_25.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/e382886f-7c7a-4045-aab3-ab3aa40c6b51_isr_1124828_25.stderr parsl.dataflow.dflow DEBUG: Task 694 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 694 submitted for App characterizeImage, waiting on task 693 parsl.dataflow.dflow DEBUG: Task 694 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 694 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 695 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 695 submitted for App calibrate, waiting on task 694 parsl.dataflow.dflow DEBUG: Task 695 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 695 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 696 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 696 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 696 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 696 has memoization hash c5756e925c2c38f9b09c5d7326ad5f01 parsl.dataflow.memoization INFO: Task 696 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 696 try 0 launched on executor multi with executor id 233 parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/4513d049-1425-42ff-ad6c-646ea7c1f5ad_isr_1124710_36.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/4513d049-1425-42ff-ad6c-646ea7c1f5ad_isr_1124710_36.stderr parsl.dataflow.dflow DEBUG: Task 697 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 697 submitted for App characterizeImage, waiting on task 696 parsl.dataflow.dflow DEBUG: Task 697 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 697 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 698 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 698 submitted for App calibrate, waiting on task 697 parsl.dataflow.dflow DEBUG: Task 698 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 698 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 699 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 699 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 699 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 699 has memoization hash 853ae586cb7b918bb61da76dcee7b57e parsl.dataflow.memoization INFO: Task 699 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 699 try 0 launched on executor multi with executor id 234 parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6b1e237f-01f4-42e0-b6e9-3bd2a7a7d69c_isr_1124704_16.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6b1e237f-01f4-42e0-b6e9-3bd2a7a7d69c_isr_1124704_16.stderr parsl.dataflow.dflow DEBUG: Task 700 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 700 submitted for App characterizeImage, waiting on task 699 parsl.dataflow.dflow DEBUG: Task 700 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 700 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 701 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 701 submitted for App calibrate, waiting on task 700 parsl.dataflow.dflow DEBUG: Task 701 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 701 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 702 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 702 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 702 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 702 has memoization hash 6065334344f9f6b7338da23af151d209 parsl.dataflow.memoization INFO: Task 702 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 702 try 0 launched on executor multi with executor id 235 parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/756b1f54-ddce-4c0a-9ce1-89c859fedd4e_isr_1124828_19.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/756b1f54-ddce-4c0a-9ce1-89c859fedd4e_isr_1124828_19.stderr parsl.dataflow.dflow DEBUG: Task 703 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 703 submitted for App characterizeImage, waiting on task 702 parsl.dataflow.dflow DEBUG: Task 703 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 703 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 704 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 704 submitted for App calibrate, waiting on task 703 parsl.dataflow.dflow DEBUG: Task 704 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 704 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 705 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 705 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 705 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 705 has memoization hash f2b837d735b808a579b759f6536e61e8 parsl.dataflow.memoization INFO: Task 705 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 705 try 0 launched on executor multi with executor id 236 parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/034fad09-72f7-4399-bed2-c264726730f8_isr_1124831_15.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/034fad09-72f7-4399-bed2-c264726730f8_isr_1124831_15.stderr parsl.dataflow.dflow DEBUG: Task 706 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 706 submitted for App characterizeImage, waiting on task 705 parsl.dataflow.dflow DEBUG: Task 706 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 706 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 707 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 707 submitted for App calibrate, waiting on task 706 parsl.dataflow.dflow DEBUG: Task 707 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 707 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 708 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 708 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 708 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 708 has memoization hash 4cdec8f8a7d1393ca9336f726193b679 parsl.dataflow.memoization INFO: Task 708 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 708 try 0 launched on executor multi with executor id 237 parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/242ebfef-de11-4667-81c6-137b91de8a84_isr_1124707_33.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/242ebfef-de11-4667-81c6-137b91de8a84_isr_1124707_33.stderr parsl.dataflow.dflow DEBUG: Task 709 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 709 submitted for App characterizeImage, waiting on task 708 parsl.dataflow.dflow DEBUG: Task 709 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 709 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 710 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 710 submitted for App calibrate, waiting on task 709 parsl.dataflow.dflow DEBUG: Task 710 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 710 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 711 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 711 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 711 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 711 has memoization hash 75b0b7321ebe1d8e0f33a15daef73b17 parsl.dataflow.memoization INFO: Task 711 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 711 try 0 launched on executor multi with executor id 238 parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a71f3239-3551-4270-943e-39907281ba25_isr_1124710_40.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a71f3239-3551-4270-943e-39907281ba25_isr_1124710_40.stderr parsl.dataflow.dflow DEBUG: Task 712 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 712 submitted for App characterizeImage, waiting on task 711 parsl.dataflow.dflow DEBUG: Task 712 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 712 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 713 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 713 submitted for App calibrate, waiting on task 712 parsl.dataflow.dflow DEBUG: Task 713 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 713 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 714 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 714 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 714 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 714 has memoization hash 77c8aa7e3fab818a49ea10572579964d parsl.dataflow.memoization INFO: Task 714 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 714 try 0 launched on executor multi with executor id 239 parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e8e468f7-575c-46ee-9b19-de61627c7e37_isr_1124704_61.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e8e468f7-575c-46ee-9b19-de61627c7e37_isr_1124704_61.stderr parsl.dataflow.dflow DEBUG: Task 715 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 715 submitted for App characterizeImage, waiting on task 714 parsl.dataflow.dflow DEBUG: Task 715 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 715 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 716 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 716 submitted for App calibrate, waiting on task 715 parsl.dataflow.dflow DEBUG: Task 716 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 716 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 717 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 717 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 717 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 717 has memoization hash 6b49b2907459030b6c5d140f9f4f46eb parsl.dataflow.memoization INFO: Task 717 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 717 try 0 launched on executor multi with executor id 240 parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c9a38688-1deb-4a36-a3e2-2a338d18bc0c_isr_1124707_13.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c9a38688-1deb-4a36-a3e2-2a338d18bc0c_isr_1124707_13.stderr parsl.dataflow.dflow DEBUG: Task 718 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 718 submitted for App characterizeImage, waiting on task 717 parsl.dataflow.dflow DEBUG: Task 718 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 718 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 719 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 719 submitted for App calibrate, waiting on task 718 parsl.dataflow.dflow DEBUG: Task 719 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 719 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 720 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 720 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 720 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 720 has memoization hash eec1c03ed57eda671a6e860237552f39 parsl.dataflow.memoization INFO: Task 720 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977c680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 720 try 0 launched on executor multi with executor id 241 parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/fd5430af-f5d7-4c1c-bc24-6ed09eb85cda_isr_1124707_44.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/fd5430af-f5d7-4c1c-bc24-6ed09eb85cda_isr_1124707_44.stderr parsl.dataflow.dflow DEBUG: Task 721 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 721 submitted for App characterizeImage, waiting on task 720 parsl.dataflow.dflow DEBUG: Task 721 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 721 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 722 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 722 submitted for App calibrate, waiting on task 721 parsl.dataflow.dflow DEBUG: Task 722 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 722 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 723 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 723 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 723 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 723 has memoization hash de5f86987175c0dbba17fee4701f49bb parsl.dataflow.memoization INFO: Task 723 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977c9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 07:25:42 proc_lsst.multi:170 [INFO] [multi] job_id 16887753 proc_lsst.multi INFO: [multi] job_id 16887753 2024-03-07 07:25:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:25:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887753 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887753 parsl.executors.status_handling DEBUG: Launched block 5 on executor multi with job ID 16887753 parsl.executors.status_handling INFO: Allocated block ID 6 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-6 2024-03-07 07:25:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-6 2024-03-07 07:25:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:25:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=6 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 723 try 0 launched on executor multi with executor id 242 parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/9a5c3e5e-7b76-4dd0-896f-aac9cd8abf5d_isr_1124704_5.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/9a5c3e5e-7b76-4dd0-896f-aac9cd8abf5d_isr_1124704_5.stderr parsl.dataflow.dflow DEBUG: Task 724 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 724 submitted for App characterizeImage, waiting on task 723 parsl.dataflow.dflow DEBUG: Task 724 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 724 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 725 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 725 submitted for App calibrate, waiting on task 724 parsl.dataflow.dflow DEBUG: Task 725 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 725 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 726 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 726 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 726 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 726 has memoization hash 98afd35ebf84fa98bc63ed3875e3d43f parsl.dataflow.memoization INFO: Task 726 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 726 try 0 launched on executor multi with executor id 243 parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/7dbbada0-cbb7-4ad9-ae4a-4c00c394e71f_isr_1124828_49.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/7dbbada0-cbb7-4ad9-ae4a-4c00c394e71f_isr_1124828_49.stderr parsl.dataflow.dflow DEBUG: Task 727 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 727 submitted for App characterizeImage, waiting on task 726 parsl.dataflow.dflow DEBUG: Task 727 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 727 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 728 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 728 submitted for App calibrate, waiting on task 727 parsl.dataflow.dflow DEBUG: Task 728 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 728 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 729 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 729 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 729 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 729 has memoization hash c7261c308acf90daa0af9c70b6d44e63 parsl.dataflow.memoization INFO: Task 729 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977ce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 729 try 0 launched on executor multi with executor id 244 parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/80f2c890-1be4-429d-a346-780b74e1f902_isr_1124831_62.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/80f2c890-1be4-429d-a346-780b74e1f902_isr_1124831_62.stderr parsl.dataflow.dflow DEBUG: Task 730 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 730 submitted for App characterizeImage, waiting on task 729 parsl.dataflow.dflow DEBUG: Task 730 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 730 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 731 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 731 submitted for App calibrate, waiting on task 730 parsl.dataflow.dflow DEBUG: Task 731 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 731 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 732 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 732 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 732 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 732 has memoization hash 5bdaea3ce4e60a5b36fa9b1c5b7524fa parsl.dataflow.memoization INFO: Task 732 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 732 try 0 launched on executor multi with executor id 245 parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/4921318f-d942-42b2-93c2-54761420e06c_isr_1124710_2.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/4921318f-d942-42b2-93c2-54761420e06c_isr_1124710_2.stderr parsl.dataflow.dflow DEBUG: Task 733 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 733 submitted for App characterizeImage, waiting on task 732 parsl.dataflow.dflow DEBUG: Task 733 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 733 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 734 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 734 submitted for App calibrate, waiting on task 733 parsl.dataflow.dflow DEBUG: Task 734 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 734 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 735 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 735 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 735 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 735 has memoization hash a8be0a07500744a775227e6ad4ee0f01 parsl.dataflow.memoization INFO: Task 735 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 735 try 0 launched on executor multi with executor id 246 parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8bde51f5-2408-4ab4-a7eb-df59df48d04e_isr_1124828_38.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8bde51f5-2408-4ab4-a7eb-df59df48d04e_isr_1124828_38.stderr parsl.dataflow.dflow DEBUG: Task 736 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 736 submitted for App characterizeImage, waiting on task 735 parsl.dataflow.dflow DEBUG: Task 736 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 736 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 737 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 737 submitted for App calibrate, waiting on task 736 parsl.dataflow.dflow DEBUG: Task 737 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 737 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 738 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 738 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 738 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 738 has memoization hash f8a991d791649d8ae1ebe967396199d6 parsl.dataflow.memoization INFO: Task 738 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 738 try 0 launched on executor multi with executor id 247 parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/cf3bfe69-1eaa-49dc-be02-13e74fb8b6fe_isr_1124707_35.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/cf3bfe69-1eaa-49dc-be02-13e74fb8b6fe_isr_1124707_35.stderr parsl.dataflow.dflow DEBUG: Task 739 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 739 submitted for App characterizeImage, waiting on task 738 parsl.dataflow.dflow DEBUG: Task 739 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 739 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 740 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 740 submitted for App calibrate, waiting on task 739 parsl.dataflow.dflow DEBUG: Task 740 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 740 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 741 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 741 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 741 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 741 has memoization hash 6554cb5d6023580a5b15155e769549af parsl.dataflow.memoization INFO: Task 741 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977d440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 741 try 0 launched on executor multi with executor id 248 parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/85c80fc7-8818-4145-b999-b08d05a0c3c9_isr_1124704_24.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/85c80fc7-8818-4145-b999-b08d05a0c3c9_isr_1124704_24.stderr parsl.dataflow.dflow DEBUG: Task 742 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 742 submitted for App characterizeImage, waiting on task 741 parsl.dataflow.dflow DEBUG: Task 742 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 742 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 743 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 743 submitted for App calibrate, waiting on task 742 parsl.dataflow.dflow DEBUG: Task 743 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 743 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 744 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 744 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 744 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 744 has memoization hash a9d33351b4356543da08e6d2e9cd2df8 parsl.dataflow.memoization INFO: Task 744 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 744 try 0 launched on executor multi with executor id 249 parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/497e5620-0715-414a-bb16-cc05c562fbb8_isr_1124831_39.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/497e5620-0715-414a-bb16-cc05c562fbb8_isr_1124831_39.stderr parsl.dataflow.dflow DEBUG: Task 745 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 745 submitted for App characterizeImage, waiting on task 744 parsl.dataflow.dflow DEBUG: Task 745 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 745 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 746 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 746 submitted for App calibrate, waiting on task 745 parsl.dataflow.dflow DEBUG: Task 746 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 746 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 747 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 747 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 747 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 747 has memoization hash d8bae23e1f842188cfac13dc593bdea7 parsl.dataflow.memoization INFO: Task 747 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 747 try 0 launched on executor multi with executor id 250 parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/82c4c721-a39f-453d-8b23-33be8c6efc29_isr_1124707_61.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/82c4c721-a39f-453d-8b23-33be8c6efc29_isr_1124707_61.stderr parsl.dataflow.dflow DEBUG: Task 748 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 748 submitted for App characterizeImage, waiting on task 747 parsl.dataflow.dflow DEBUG: Task 748 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 748 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 749 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 749 submitted for App calibrate, waiting on task 748 parsl.dataflow.dflow DEBUG: Task 749 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 749 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 750 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 750 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 750 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 750 has memoization hash b2543ca82205761a294949e4b4eacc89 parsl.dataflow.memoization INFO: Task 750 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 750 try 0 launched on executor multi with executor id 251 parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/692bdb39-2274-4e50-a2f1-2c606e8edbab_isr_1124831_11.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/692bdb39-2274-4e50-a2f1-2c606e8edbab_isr_1124831_11.stderr parsl.dataflow.dflow DEBUG: Task 751 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 751 submitted for App characterizeImage, waiting on task 750 parsl.dataflow.dflow DEBUG: Task 751 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 751 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 752 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 752 submitted for App calibrate, waiting on task 751 parsl.dataflow.dflow DEBUG: Task 752 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 752 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 753 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 753 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 753 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 753 has memoization hash 6d43ff39b34515394b8a4880338072d3 parsl.dataflow.memoization INFO: Task 753 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 753 try 0 launched on executor multi with executor id 252 parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/0d5d7c77-f177-4adc-bc18-f2e4bb1cf767_isr_1124704_38.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/0d5d7c77-f177-4adc-bc18-f2e4bb1cf767_isr_1124704_38.stderr parsl.dataflow.dflow DEBUG: Task 754 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 754 submitted for App characterizeImage, waiting on task 753 parsl.dataflow.dflow DEBUG: Task 754 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 754 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 755 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 755 submitted for App calibrate, waiting on task 754 parsl.dataflow.dflow DEBUG: Task 755 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 755 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 756 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 756 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 756 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 756 has memoization hash 9c2437b7c55a5e7790b8cefabc086e3f parsl.dataflow.memoization INFO: Task 756 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 756 try 0 launched on executor multi with executor id 253 parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/96bc151b-db8b-4c47-bfbc-5ef65354fbfd_isr_1124707_7.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/96bc151b-db8b-4c47-bfbc-5ef65354fbfd_isr_1124707_7.stderr parsl.dataflow.dflow DEBUG: Task 757 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 757 submitted for App characterizeImage, waiting on task 756 parsl.dataflow.dflow DEBUG: Task 757 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 757 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 758 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 758 submitted for App calibrate, waiting on task 757 parsl.dataflow.dflow DEBUG: Task 758 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 758 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 759 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 759 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 759 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 759 has memoization hash 70dfa67ef58591cd775c9e0ab5175bd4 parsl.dataflow.memoization INFO: Task 759 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 759 try 0 launched on executor multi with executor id 254 parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/076460fd-ba33-42ad-a636-79f62af670f3_isr_1124704_51.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/076460fd-ba33-42ad-a636-79f62af670f3_isr_1124704_51.stderr parsl.dataflow.dflow DEBUG: Task 760 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 760 submitted for App characterizeImage, waiting on task 759 parsl.dataflow.dflow DEBUG: Task 760 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 760 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 761 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 761 submitted for App calibrate, waiting on task 760 parsl.dataflow.dflow DEBUG: Task 761 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 761 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 762 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 762 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 762 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 762 has memoization hash 65b97aca65951d91408d41b5b4a97668 parsl.dataflow.memoization INFO: Task 762 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 762 try 0 launched on executor multi with executor id 255 parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/5e753b90-61a3-4aa5-a648-f2a32e4d2925_isr_1124828_40.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/5e753b90-61a3-4aa5-a648-f2a32e4d2925_isr_1124828_40.stderr parsl.dataflow.dflow DEBUG: Task 763 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 763 submitted for App characterizeImage, waiting on task 762 parsl.dataflow.dflow DEBUG: Task 763 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 763 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 764 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 764 submitted for App calibrate, waiting on task 763 parsl.dataflow.dflow DEBUG: Task 764 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 764 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 765 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 765 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 765 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 765 has memoization hash 966c6b6c7d6642062921e2fe3f570ec6 parsl.dataflow.memoization INFO: Task 765 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977ed40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 765 try 0 launched on executor multi with executor id 256 parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/0f0a93d9-5b14-4ef1-bbec-dc820aa026dd_isr_1124704_31.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/0f0a93d9-5b14-4ef1-bbec-dc820aa026dd_isr_1124704_31.stderr parsl.dataflow.dflow DEBUG: Task 766 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 766 submitted for App characterizeImage, waiting on task 765 parsl.dataflow.dflow DEBUG: Task 766 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 766 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 767 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 767 submitted for App calibrate, waiting on task 766 parsl.dataflow.dflow DEBUG: Task 767 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 767 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 768 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 768 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 768 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 768 has memoization hash 21d2f6fd4b205ffe4fefc902d19541ef parsl.dataflow.memoization INFO: Task 768 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 768 try 0 launched on executor multi with executor id 257 parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/c1e93f2a-b4f9-443c-b8bd-802a7f0cfc5c_isr_1124704_59.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/c1e93f2a-b4f9-443c-b8bd-802a7f0cfc5c_isr_1124704_59.stderr parsl.dataflow.dflow DEBUG: Task 769 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 769 submitted for App characterizeImage, waiting on task 768 parsl.dataflow.dflow DEBUG: Task 769 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 769 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 770 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 770 submitted for App calibrate, waiting on task 769 parsl.dataflow.dflow DEBUG: Task 770 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 770 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 771 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 771 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 771 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 771 has memoization hash 547e279e91c7f6b6d4ed65d7a662ffa6 parsl.dataflow.memoization INFO: Task 771 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 771 try 0 launched on executor multi with executor id 258 parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/12179eb3-1fb3-4168-80c0-4a46c1d1932c_isr_1124831_2.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/12179eb3-1fb3-4168-80c0-4a46c1d1932c_isr_1124831_2.stderr parsl.dataflow.dflow DEBUG: Task 772 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 772 submitted for App characterizeImage, waiting on task 771 parsl.dataflow.dflow DEBUG: Task 772 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 772 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 773 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 773 submitted for App calibrate, waiting on task 772 parsl.dataflow.dflow DEBUG: Task 773 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 773 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 774 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 774 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 774 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 774 has memoization hash a39061f13ef9f9f1416b6e4b9453a497 parsl.dataflow.memoization INFO: Task 774 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 774 try 0 launched on executor multi with executor id 259 parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/0cad93c9-ab9f-46de-a094-cad87202ff82_isr_1124710_8.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/0cad93c9-ab9f-46de-a094-cad87202ff82_isr_1124710_8.stderr parsl.dataflow.dflow DEBUG: Task 775 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 775 submitted for App characterizeImage, waiting on task 774 parsl.dataflow.dflow DEBUG: Task 775 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 775 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 776 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 776 submitted for App calibrate, waiting on task 775 parsl.dataflow.dflow DEBUG: Task 776 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 776 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 777 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 777 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 777 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 777 has memoization hash d1aff978494c26599f1a5441054e6fd3 parsl.dataflow.memoization INFO: Task 777 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 777 try 0 launched on executor multi with executor id 260 parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/5f7cc30b-538c-4995-8110-58fdd6eaa9b9_isr_1124828_3.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/5f7cc30b-538c-4995-8110-58fdd6eaa9b9_isr_1124828_3.stderr parsl.dataflow.dflow DEBUG: Task 778 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 778 submitted for App characterizeImage, waiting on task 777 parsl.dataflow.dflow DEBUG: Task 778 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 778 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 779 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 779 submitted for App calibrate, waiting on task 778 parsl.dataflow.dflow DEBUG: Task 779 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 779 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 780 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 780 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 780 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 780 has memoization hash 05994e2d182a07693677016f45312a48 parsl.dataflow.memoization INFO: Task 780 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 780 try 0 launched on executor multi with executor id 261 parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/5ff1fe47-fda0-4675-955d-4e3e8c8132be_isr_1124704_52.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/5ff1fe47-fda0-4675-955d-4e3e8c8132be_isr_1124704_52.stderr parsl.dataflow.dflow DEBUG: Task 781 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 781 submitted for App characterizeImage, waiting on task 780 parsl.dataflow.dflow DEBUG: Task 781 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 781 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 782 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 782 submitted for App calibrate, waiting on task 781 parsl.dataflow.dflow DEBUG: Task 782 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 782 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 783 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 783 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 783 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 783 has memoization hash 186dfc876febe3212f1be1d80843b5dd parsl.dataflow.memoization INFO: Task 783 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 783 try 0 launched on executor multi with executor id 262 parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f62bc6f4-b871-42e3-96ab-9ca802bb072d_isr_1124710_20.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f62bc6f4-b871-42e3-96ab-9ca802bb072d_isr_1124710_20.stderr parsl.dataflow.dflow DEBUG: Task 784 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 784 submitted for App characterizeImage, waiting on task 783 parsl.dataflow.dflow DEBUG: Task 784 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 784 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 785 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 785 submitted for App calibrate, waiting on task 784 parsl.dataflow.dflow DEBUG: Task 785 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 785 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 786 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 786 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 786 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 786 has memoization hash 28f70f4b086de8531ac8c05ae60b7fa6 parsl.dataflow.memoization INFO: Task 786 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 786 try 0 launched on executor multi with executor id 263 parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bf4f389f-20dc-4c41-a679-3f9dd5e9244f_isr_1124828_5.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bf4f389f-20dc-4c41-a679-3f9dd5e9244f_isr_1124828_5.stderr parsl.dataflow.dflow DEBUG: Task 787 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 787 submitted for App characterizeImage, waiting on task 786 parsl.dataflow.dflow DEBUG: Task 787 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 787 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 788 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 788 submitted for App calibrate, waiting on task 787 parsl.dataflow.dflow DEBUG: Task 788 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 788 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 789 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 789 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 789 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 789 has memoization hash b5eb45fb52da4cfc21327b42b54d2de2 parsl.dataflow.memoization INFO: Task 789 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 07:25:42 proc_lsst.multi:170 [INFO] [multi] job_id 16887754 proc_lsst.multi INFO: [multi] job_id 16887754 2024-03-07 07:25:42 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 07:25:42 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:42 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887754 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887754 parsl.executors.status_handling DEBUG: Launched block 6 on executor multi with job ID 16887754 parsl.executors.status_handling INFO: Allocated block ID 7 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-7 2024-03-07 07:25:42 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-7 2024-03-07 07:25:42 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:42 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:42 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 07:25:42 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:42 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=7 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 789 try 0 launched on executor multi with executor id 264 parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/be0a15b7-d0bd-4356-8a07-6b064a480bac_isr_1124704_47.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/be0a15b7-d0bd-4356-8a07-6b064a480bac_isr_1124704_47.stderr parsl.dataflow.dflow DEBUG: Task 790 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 790 submitted for App characterizeImage, waiting on task 789 parsl.dataflow.dflow DEBUG: Task 790 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 790 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 791 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 791 submitted for App calibrate, waiting on task 790 parsl.dataflow.dflow DEBUG: Task 791 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 791 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 792 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 792 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 792 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 792 has memoization hash 97015cfbe4a098943862e7b19f7039ac parsl.dataflow.memoization INFO: Task 792 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 792 try 0 launched on executor multi with executor id 265 parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/3f5d340a-a590-4ea2-82de-da5d026fe97a_isr_1124707_45.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/3f5d340a-a590-4ea2-82de-da5d026fe97a_isr_1124707_45.stderr parsl.dataflow.dflow DEBUG: Task 793 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 793 submitted for App characterizeImage, waiting on task 792 parsl.dataflow.dflow DEBUG: Task 793 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 793 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 794 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 794 submitted for App calibrate, waiting on task 793 parsl.dataflow.dflow DEBUG: Task 794 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 794 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 795 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 795 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 795 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 795 has memoization hash 5862c5eb1df8306a6252273f81f21499 parsl.dataflow.memoization INFO: Task 795 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c9300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 795 try 0 launched on executor multi with executor id 266 parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/e4ba0759-e842-43ed-8b4e-8c5d0417871c_isr_1124710_13.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/e4ba0759-e842-43ed-8b4e-8c5d0417871c_isr_1124710_13.stderr parsl.dataflow.dflow DEBUG: Task 796 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 796 submitted for App characterizeImage, waiting on task 795 parsl.dataflow.dflow DEBUG: Task 796 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 796 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 797 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 797 submitted for App calibrate, waiting on task 796 parsl.dataflow.dflow DEBUG: Task 797 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 797 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 798 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 798 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 798 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 798 has memoization hash 5b36b2a7ce497f8cb8ae74acbabb1699 parsl.dataflow.memoization INFO: Task 798 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 798 try 0 launched on executor multi with executor id 267 parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a73d3153-97ce-4edc-9720-8ab4ab9b77b4_isr_1124710_7.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a73d3153-97ce-4edc-9720-8ab4ab9b77b4_isr_1124710_7.stderr parsl.dataflow.dflow DEBUG: Task 799 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 799 submitted for App characterizeImage, waiting on task 798 parsl.dataflow.dflow DEBUG: Task 799 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 799 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 800 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 800 submitted for App calibrate, waiting on task 799 parsl.dataflow.dflow DEBUG: Task 800 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 800 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 801 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 801 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 801 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 801 has memoization hash d99defbc97ae936b6322fa878b667aec parsl.dataflow.memoization INFO: Task 801 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cbce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 801 try 0 launched on executor multi with executor id 268 parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/c84f2cd4-6afd-4488-80f8-da21ebdffa4e_isr_1124831_41.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/c84f2cd4-6afd-4488-80f8-da21ebdffa4e_isr_1124831_41.stderr parsl.dataflow.dflow DEBUG: Task 802 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 802 submitted for App characterizeImage, waiting on task 801 parsl.dataflow.dflow DEBUG: Task 802 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 802 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 803 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 803 submitted for App calibrate, waiting on task 802 parsl.dataflow.dflow DEBUG: Task 803 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 803 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 804 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 804 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 804 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 804 has memoization hash 4e9199a80489cb5d4375f9488a32978f parsl.dataflow.memoization INFO: Task 804 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 804 try 0 launched on executor multi with executor id 269 parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/77bba865-fe20-4808-84aa-7fb6fd672671_isr_1124828_4.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/77bba865-fe20-4808-84aa-7fb6fd672671_isr_1124828_4.stderr parsl.dataflow.dflow DEBUG: Task 805 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 805 submitted for App characterizeImage, waiting on task 804 parsl.dataflow.dflow DEBUG: Task 805 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 805 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 806 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 806 submitted for App calibrate, waiting on task 805 parsl.dataflow.dflow DEBUG: Task 806 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 806 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 807 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 807 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 807 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 807 has memoization hash ce3a6eeb951cbe33d44df365eb6a1e5d parsl.dataflow.memoization INFO: Task 807 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296080e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 807 try 0 launched on executor multi with executor id 270 parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/32b5c052-efbf-435e-90d6-c1ed903f73d5_isr_1124828_57.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/32b5c052-efbf-435e-90d6-c1ed903f73d5_isr_1124828_57.stderr parsl.dataflow.dflow DEBUG: Task 808 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 808 submitted for App characterizeImage, waiting on task 807 parsl.dataflow.dflow DEBUG: Task 808 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 808 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 809 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 809 submitted for App calibrate, waiting on task 808 parsl.dataflow.dflow DEBUG: Task 809 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 809 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 810 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 810 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 810 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 810 has memoization hash b5d76d8d08a8dd476b69232aabbd50a1 parsl.dataflow.memoization INFO: Task 810 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 810 try 0 launched on executor multi with executor id 271 parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bb53a21b-8b5f-432b-b7da-178d3e3a7d4d_isr_1124828_6.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bb53a21b-8b5f-432b-b7da-178d3e3a7d4d_isr_1124828_6.stderr parsl.dataflow.dflow DEBUG: Task 811 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 811 submitted for App characterizeImage, waiting on task 810 parsl.dataflow.dflow DEBUG: Task 811 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 811 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 812 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 812 submitted for App calibrate, waiting on task 811 parsl.dataflow.dflow DEBUG: Task 812 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 812 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 813 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 813 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 813 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 813 has memoization hash 7d2fe7ba6d429648c0be2ab6af685b75 parsl.dataflow.memoization INFO: Task 813 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c9d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 813 try 0 launched on executor multi with executor id 272 parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/49c4cc2d-d1a7-48a5-b87d-24198533e99c_isr_1124710_38.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/49c4cc2d-d1a7-48a5-b87d-24198533e99c_isr_1124710_38.stderr parsl.dataflow.dflow DEBUG: Task 814 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 814 submitted for App characterizeImage, waiting on task 813 parsl.dataflow.dflow DEBUG: Task 814 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 814 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 815 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 815 submitted for App calibrate, waiting on task 814 parsl.dataflow.dflow DEBUG: Task 815 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 815 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 816 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 816 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 816 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 816 has memoization hash 9d2fca43e569bc960bc5ce6f673ae6c5 parsl.dataflow.memoization INFO: Task 816 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295ca0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 816 try 0 launched on executor multi with executor id 273 parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f800fbbf-7199-4a7d-9098-ab1f809652c2_isr_1124831_17.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f800fbbf-7199-4a7d-9098-ab1f809652c2_isr_1124831_17.stderr parsl.dataflow.dflow DEBUG: Task 817 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 817 submitted for App characterizeImage, waiting on task 816 parsl.dataflow.dflow DEBUG: Task 817 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 817 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 818 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 818 submitted for App calibrate, waiting on task 817 parsl.dataflow.dflow DEBUG: Task 818 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 818 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 819 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 819 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 819 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 819 has memoization hash e1e60295c4c54b05dc79a6eb638729a7 parsl.dataflow.memoization INFO: Task 819 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295ca200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 819 try 0 launched on executor multi with executor id 274 parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/00c6d61c-c4e2-48d4-b536-99ffeb7cb84c_isr_1124707_52.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/00c6d61c-c4e2-48d4-b536-99ffeb7cb84c_isr_1124707_52.stderr parsl.dataflow.dflow DEBUG: Task 820 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 820 submitted for App characterizeImage, waiting on task 819 parsl.dataflow.dflow DEBUG: Task 820 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 820 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 821 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 821 submitted for App calibrate, waiting on task 820 parsl.dataflow.dflow DEBUG: Task 821 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 821 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 822 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 822 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 822 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 822 has memoization hash ab08e854f1a16c9d0c05c5ff80132a6e parsl.dataflow.memoization INFO: Task 822 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295ca520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 822 try 0 launched on executor multi with executor id 275 parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/deed4ecc-346b-44ea-8b02-774f822aa767_isr_1124831_12.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/deed4ecc-346b-44ea-8b02-774f822aa767_isr_1124831_12.stderr parsl.dataflow.dflow DEBUG: Task 823 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 823 submitted for App characterizeImage, waiting on task 822 parsl.dataflow.dflow DEBUG: Task 823 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 823 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 824 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 824 submitted for App calibrate, waiting on task 823 parsl.dataflow.dflow DEBUG: Task 824 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 824 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 825 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 825 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 825 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 825 has memoization hash d653ccb16b1760810804ee4e36614527 parsl.dataflow.memoization INFO: Task 825 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295ca980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 825 try 0 launched on executor multi with executor id 276 parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/5832ed04-5208-4c94-a871-866fbf40aa48_isr_1124704_32.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/5832ed04-5208-4c94-a871-866fbf40aa48_isr_1124704_32.stderr parsl.dataflow.dflow DEBUG: Task 826 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 826 submitted for App characterizeImage, waiting on task 825 parsl.dataflow.dflow DEBUG: Task 826 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 826 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 827 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 827 submitted for App calibrate, waiting on task 826 parsl.dataflow.dflow DEBUG: Task 827 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 827 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 828 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 828 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 828 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 828 has memoization hash 935335151d379fa62e99047fd22c93ef parsl.dataflow.memoization INFO: Task 828 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295caca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 828 try 0 launched on executor multi with executor id 277 parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/0c727113-2835-4683-b1d8-55bd0c235af0_isr_1124828_56.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/0c727113-2835-4683-b1d8-55bd0c235af0_isr_1124828_56.stderr parsl.dataflow.dflow DEBUG: Task 829 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 829 submitted for App characterizeImage, waiting on task 828 parsl.dataflow.dflow DEBUG: Task 829 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 829 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 830 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 830 submitted for App calibrate, waiting on task 829 parsl.dataflow.dflow DEBUG: Task 830 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 830 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 831 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 831 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 831 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 831 has memoization hash ff79567559828ebffb86c750a9d45cbb parsl.dataflow.memoization INFO: Task 831 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295caf20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 831 try 0 launched on executor multi with executor id 278 parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/8a74add7-1c93-4b1f-88b7-1bb992c8c5fe_isr_1124704_13.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/8a74add7-1c93-4b1f-88b7-1bb992c8c5fe_isr_1124704_13.stderr parsl.dataflow.dflow DEBUG: Task 832 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 832 submitted for App characterizeImage, waiting on task 831 parsl.dataflow.dflow DEBUG: Task 832 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 832 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 833 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 833 submitted for App calibrate, waiting on task 832 parsl.dataflow.dflow DEBUG: Task 833 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 833 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 834 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 834 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 834 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 834 has memoization hash d38121917c4e7dbca667cd60fdb0a2a8 parsl.dataflow.memoization INFO: Task 834 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 834 try 0 launched on executor multi with executor id 279 parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/1541a5d5-0501-4ca6-9db1-5abe744bd9e6_isr_1124707_57.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/1541a5d5-0501-4ca6-9db1-5abe744bd9e6_isr_1124707_57.stderr parsl.dataflow.dflow DEBUG: Task 835 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 835 submitted for App characterizeImage, waiting on task 834 parsl.dataflow.dflow DEBUG: Task 835 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 835 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 836 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 836 submitted for App calibrate, waiting on task 835 parsl.dataflow.dflow DEBUG: Task 836 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 836 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 837 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 837 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 837 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 837 has memoization hash 5c782002e54dcd8c35fb740f7e4da654 parsl.dataflow.memoization INFO: Task 837 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 837 try 0 launched on executor multi with executor id 280 parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/3ed84469-5fcd-4e63-8d32-84940fa60bf5_isr_1124710_14.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/3ed84469-5fcd-4e63-8d32-84940fa60bf5_isr_1124710_14.stderr parsl.dataflow.dflow DEBUG: Task 838 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 838 submitted for App characterizeImage, waiting on task 837 parsl.dataflow.dflow DEBUG: Task 838 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 838 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 839 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 839 submitted for App calibrate, waiting on task 838 parsl.dataflow.dflow DEBUG: Task 839 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 839 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 840 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 840 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 840 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 840 has memoization hash f50f387e0fbdb19a82bdc41eed228f32 parsl.dataflow.memoization INFO: Task 840 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 840 try 0 launched on executor multi with executor id 281 parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/1ac21610-c6b5-4142-9def-ac7ba86c4d15_isr_1124828_39.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/1ac21610-c6b5-4142-9def-ac7ba86c4d15_isr_1124828_39.stderr parsl.dataflow.dflow DEBUG: Task 841 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 841 submitted for App characterizeImage, waiting on task 840 parsl.dataflow.dflow DEBUG: Task 841 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 841 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 842 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 842 submitted for App calibrate, waiting on task 841 parsl.dataflow.dflow DEBUG: Task 842 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 842 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 843 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 843 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 843 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 843 has memoization hash fc329a7e991f999525da436c84c711e8 parsl.dataflow.memoization INFO: Task 843 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 843 try 0 launched on executor multi with executor id 282 parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/ddb1a578-9592-4d55-bda1-ea474f0f2c4e_isr_1124707_62.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/ddb1a578-9592-4d55-bda1-ea474f0f2c4e_isr_1124707_62.stderr parsl.dataflow.dflow DEBUG: Task 844 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 844 submitted for App characterizeImage, waiting on task 843 parsl.dataflow.dflow DEBUG: Task 844 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 844 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 845 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 845 submitted for App calibrate, waiting on task 844 parsl.dataflow.dflow DEBUG: Task 845 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 845 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 846 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 846 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 846 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 846 has memoization hash 2d5bfab313b0229dd47f31054995b3d3 parsl.dataflow.memoization INFO: Task 846 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 846 try 0 launched on executor multi with executor id 283 parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0e6c25f1-00d9-42b7-9c97-de5fd602e977_isr_1124707_59.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0e6c25f1-00d9-42b7-9c97-de5fd602e977_isr_1124707_59.stderr parsl.dataflow.dflow DEBUG: Task 847 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 847 submitted for App characterizeImage, waiting on task 846 parsl.dataflow.dflow DEBUG: Task 847 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 847 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 848 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 848 submitted for App calibrate, waiting on task 847 parsl.dataflow.dflow DEBUG: Task 848 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 848 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 849 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 849 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 849 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 849 has memoization hash c94c83602169a1ad284392725f3f2449 parsl.dataflow.memoization INFO: Task 849 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 849 try 0 launched on executor multi with executor id 284 parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/2cb0b756-b53b-4381-b683-45f26c556322_isr_1124828_8.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/2cb0b756-b53b-4381-b683-45f26c556322_isr_1124828_8.stderr parsl.dataflow.dflow DEBUG: Task 850 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 850 submitted for App characterizeImage, waiting on task 849 parsl.dataflow.dflow DEBUG: Task 850 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 850 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 851 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 851 submitted for App calibrate, waiting on task 850 parsl.dataflow.dflow DEBUG: Task 851 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 851 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 852 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 852 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 852 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 852 has memoization hash 720b58d6abfeb578e0b1318a67f9c29c parsl.dataflow.memoization INFO: Task 852 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529609080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 852 try 0 launched on executor multi with executor id 285 parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/c6722b37-a97c-45f8-986c-2405f8777f44_isr_1124828_44.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/c6722b37-a97c-45f8-986c-2405f8777f44_isr_1124828_44.stderr parsl.dataflow.dflow DEBUG: Task 853 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 853 submitted for App characterizeImage, waiting on task 852 parsl.dataflow.dflow DEBUG: Task 853 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 853 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 854 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 854 submitted for App calibrate, waiting on task 853 parsl.dataflow.dflow DEBUG: Task 854 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 854 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 855 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 855 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 855 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 855 has memoization hash ee9c0f38279566567f0a824fcfa96721 parsl.dataflow.memoization INFO: Task 855 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296093a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 855 try 0 launched on executor multi with executor id 286 parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f26304c3-ba6a-49d3-bde4-01f1b6ffb869_isr_1124710_5.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f26304c3-ba6a-49d3-bde4-01f1b6ffb869_isr_1124710_5.stderr parsl.dataflow.dflow DEBUG: Task 856 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 856 submitted for App characterizeImage, waiting on task 855 parsl.dataflow.dflow DEBUG: Task 856 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 856 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 857 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 857 submitted for App calibrate, waiting on task 856 parsl.dataflow.dflow DEBUG: Task 857 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 857 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 858 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 858 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 858 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 858 has memoization hash 1f1906ef35ef368f33d70cd7d715ef4a parsl.dataflow.memoization INFO: Task 858 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296096c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 858 try 0 launched on executor multi with executor id 287 parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/2b086f0a-3d53-4043-a10b-a60ecac1ed26_isr_1124828_52.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/2b086f0a-3d53-4043-a10b-a60ecac1ed26_isr_1124828_52.stderr parsl.dataflow.dflow DEBUG: Task 859 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 859 submitted for App characterizeImage, waiting on task 858 parsl.dataflow.dflow DEBUG: Task 859 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 859 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 860 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 860 submitted for App calibrate, waiting on task 859 parsl.dataflow.dflow DEBUG: Task 860 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 860 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 861 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 861 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 861 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 861 has memoization hash f4bd283226ef672be18fa7d86bd80241 parsl.dataflow.memoization INFO: Task 861 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296099e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 861 try 0 launched on executor multi with executor id 288 parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/8fe87486-8b36-4971-aa3f-3cff0082839f_isr_1124707_28.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/8fe87486-8b36-4971-aa3f-3cff0082839f_isr_1124707_28.stderr parsl.dataflow.dflow DEBUG: Task 862 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 862 submitted for App characterizeImage, waiting on task 861 parsl.dataflow.dflow DEBUG: Task 862 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 862 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 863 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 863 submitted for App calibrate, waiting on task 862 parsl.dataflow.dflow DEBUG: Task 863 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 863 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 864 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 864 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 864 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 864 has memoization hash 41cff2039bedb47c2ba5c34ca40c99bd parsl.dataflow.memoization INFO: Task 864 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529609d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 864 try 0 launched on executor multi with executor id 289 parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/80ea5f0c-0c6d-4101-acf9-d5e6cd71af36_isr_1124707_14.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/80ea5f0c-0c6d-4101-acf9-d5e6cd71af36_isr_1124707_14.stderr parsl.dataflow.dflow DEBUG: Task 865 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 865 submitted for App characterizeImage, waiting on task 864 parsl.dataflow.dflow DEBUG: Task 865 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 865 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 866 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 866 submitted for App calibrate, waiting on task 865 parsl.dataflow.dflow DEBUG: Task 866 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 866 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 867 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 867 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 867 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 867 has memoization hash 8ffffeff77e868cb39c08f3e0a8b68d1 parsl.dataflow.memoization INFO: Task 867 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 867 try 0 launched on executor multi with executor id 290 parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/66c7eb2d-52f5-4164-8e36-58e4e7fa342b_isr_1124831_26.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/66c7eb2d-52f5-4164-8e36-58e4e7fa342b_isr_1124831_26.stderr parsl.dataflow.dflow DEBUG: Task 868 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 868 submitted for App characterizeImage, waiting on task 867 parsl.dataflow.dflow DEBUG: Task 868 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 868 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 869 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 869 submitted for App calibrate, waiting on task 868 parsl.dataflow.dflow DEBUG: Task 869 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 869 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 870 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 870 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 870 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 870 has memoization hash 5fa7efe08c013be1f6d956c339913660 parsl.dataflow.memoization INFO: Task 870 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 870 try 0 launched on executor multi with executor id 291 parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/5a7ab523-21ef-4720-a8e5-f595b8926e65_isr_1124704_15.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/5a7ab523-21ef-4720-a8e5-f595b8926e65_isr_1124704_15.stderr parsl.dataflow.dflow DEBUG: Task 871 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 871 submitted for App characterizeImage, waiting on task 870 parsl.dataflow.dflow DEBUG: Task 871 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 871 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 872 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 872 submitted for App calibrate, waiting on task 871 parsl.dataflow.dflow DEBUG: Task 872 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 872 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 873 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 873 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 873 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 873 has memoization hash 6769418f65373ab1b5f88e4a0f065d38 parsl.dataflow.memoization INFO: Task 873 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960a660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 873 try 0 launched on executor multi with executor id 292 parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f1dc2506-5c18-4a32-957b-a4c20890cdb8_isr_1124831_38.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f1dc2506-5c18-4a32-957b-a4c20890cdb8_isr_1124831_38.stderr parsl.dataflow.dflow DEBUG: Task 874 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 874 submitted for App characterizeImage, waiting on task 873 parsl.dataflow.dflow DEBUG: Task 874 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 874 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 875 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 875 submitted for App calibrate, waiting on task 874 parsl.dataflow.dflow DEBUG: Task 875 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 875 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 876 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 876 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 876 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 876 has memoization hash 7c39bddcfeee0996b2338b1ed8a5c2e6 parsl.dataflow.memoization INFO: Task 876 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960a980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 876 try 0 launched on executor multi with executor id 293 parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/37aaf610-18a5-4cdd-a185-b43016359892_isr_1124828_12.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/37aaf610-18a5-4cdd-a185-b43016359892_isr_1124828_12.stderr parsl.dataflow.dflow DEBUG: Task 877 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 877 submitted for App characterizeImage, waiting on task 876 parsl.dataflow.dflow DEBUG: Task 877 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 877 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 878 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 878 submitted for App calibrate, waiting on task 877 parsl.dataflow.dflow DEBUG: Task 878 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 878 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 879 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 879 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 879 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 879 has memoization hash e92cdb14f524aa9f7d18410a1e87e44b parsl.dataflow.memoization INFO: Task 879 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960aca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 879 try 0 launched on executor multi with executor id 294 parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/05d6d796-05b1-4997-bce8-8064a3cf8ace_isr_1124704_54.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/05d6d796-05b1-4997-bce8-8064a3cf8ace_isr_1124704_54.stderr parsl.dataflow.dflow DEBUG: Task 880 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 880 submitted for App characterizeImage, waiting on task 879 parsl.dataflow.dflow DEBUG: Task 880 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 880 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 881 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 881 submitted for App calibrate, waiting on task 880 parsl.dataflow.dflow DEBUG: Task 881 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 881 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 882 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 882 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 882 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 882 has memoization hash e48869f8111427c6044fdff576f709c0 parsl.dataflow.memoization INFO: Task 882 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960afc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 882 try 0 launched on executor multi with executor id 295 parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/fe76f12c-f0e7-4a74-bca4-80910896cbaf_isr_1124828_22.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/fe76f12c-f0e7-4a74-bca4-80910896cbaf_isr_1124828_22.stderr parsl.dataflow.dflow DEBUG: Task 883 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 883 submitted for App characterizeImage, waiting on task 882 parsl.dataflow.dflow DEBUG: Task 883 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 883 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 884 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 884 submitted for App calibrate, waiting on task 883 parsl.dataflow.dflow DEBUG: Task 884 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 884 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 885 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 885 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 885 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 885 has memoization hash 54229a8d2f3535d951e037d6e371f12c parsl.dataflow.memoization INFO: Task 885 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960b2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 885 try 0 launched on executor multi with executor id 296 parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/14b5c581-1ad6-4c8c-9606-6ff755abe346_isr_1124704_57.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/14b5c581-1ad6-4c8c-9606-6ff755abe346_isr_1124704_57.stderr parsl.dataflow.dflow DEBUG: Task 886 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 886 submitted for App characterizeImage, waiting on task 885 parsl.dataflow.dflow DEBUG: Task 886 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 886 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 887 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 887 submitted for App calibrate, waiting on task 886 parsl.dataflow.dflow DEBUG: Task 887 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 887 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 888 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 888 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 888 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 888 has memoization hash 075de847a088d1544908ba1a16b05fae parsl.dataflow.memoization INFO: Task 888 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960b600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 888 try 0 launched on executor multi with executor id 297 parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/172428d8-8195-41f4-8c07-9d0abe0f180a_isr_1124707_29.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/172428d8-8195-41f4-8c07-9d0abe0f180a_isr_1124707_29.stderr parsl.dataflow.dflow DEBUG: Task 889 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 889 submitted for App characterizeImage, waiting on task 888 parsl.dataflow.dflow DEBUG: Task 889 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 889 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 890 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 890 submitted for App calibrate, waiting on task 889 parsl.dataflow.dflow DEBUG: Task 890 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 890 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 891 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 891 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 891 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 891 has memoization hash 1368a621a13bd3cdf4bf368d6c4e40c7 parsl.dataflow.memoization INFO: Task 891 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960b920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 891 try 0 launched on executor multi with executor id 298 parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/21aa76a4-fc66-435a-bb0a-2f29fb959cd8_isr_1124707_23.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/21aa76a4-fc66-435a-bb0a-2f29fb959cd8_isr_1124707_23.stderr parsl.dataflow.dflow DEBUG: Task 892 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 892 submitted for App characterizeImage, waiting on task 891 parsl.dataflow.dflow DEBUG: Task 892 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 892 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 893 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 893 submitted for App calibrate, waiting on task 892 parsl.dataflow.dflow DEBUG: Task 893 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 893 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 894 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 894 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 894 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 894 has memoization hash dc043fc84ee59d02fdc0aa0d89b49e57 parsl.dataflow.memoization INFO: Task 894 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960bc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 894 try 0 launched on executor multi with executor id 299 parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/2ffa1e49-dda0-4148-a2f4-b68186242566_isr_1124710_59.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/2ffa1e49-dda0-4148-a2f4-b68186242566_isr_1124710_59.stderr parsl.dataflow.dflow DEBUG: Task 895 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 895 submitted for App characterizeImage, waiting on task 894 parsl.dataflow.dflow DEBUG: Task 895 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 895 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 896 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 896 submitted for App calibrate, waiting on task 895 parsl.dataflow.dflow DEBUG: Task 896 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 896 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 897 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 897 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 897 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 897 has memoization hash 4c4f5537bcf316765cb14f4d26574fe2 parsl.dataflow.memoization INFO: Task 897 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 897 try 0 launched on executor multi with executor id 300 parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/500bc128-6b2d-4af7-8652-1b9acc85bfe6_isr_1124704_44.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/500bc128-6b2d-4af7-8652-1b9acc85bfe6_isr_1124704_44.stderr parsl.dataflow.dflow DEBUG: Task 898 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 898 submitted for App characterizeImage, waiting on task 897 parsl.dataflow.dflow DEBUG: Task 898 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 898 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 899 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 899 submitted for App calibrate, waiting on task 898 parsl.dataflow.dflow DEBUG: Task 899 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 899 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 900 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 900 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 900 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 900 has memoization hash a44b5ec5f8dd4d9c413bfbbae8f043e7 parsl.dataflow.memoization INFO: Task 900 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296642c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 07:25:43 proc_lsst.multi:170 [INFO] [multi] job_id 16887755 proc_lsst.multi INFO: [multi] job_id 16887755 2024-03-07 07:25:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 07:25:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887755 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887755 parsl.executors.status_handling DEBUG: Launched block 7 on executor multi with job ID 16887755 parsl.executors.status_handling INFO: Allocated block ID 8 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-8 2024-03-07 07:25:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-8 2024-03-07 07:25:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 07:25:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=8 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Parsl task 900 try 0 launched on executor multi with executor id 301 parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/47612486-9fdd-4376-85f2-7cef0c982bfa_isr_1124704_23.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/47612486-9fdd-4376-85f2-7cef0c982bfa_isr_1124704_23.stderr parsl.dataflow.dflow DEBUG: Task 901 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 901 submitted for App characterizeImage, waiting on task 900 parsl.dataflow.dflow DEBUG: Task 901 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 901 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 902 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 902 submitted for App calibrate, waiting on task 901 parsl.dataflow.dflow DEBUG: Task 902 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 902 has outstanding dependencies, so launch_if_ready skipping parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 903 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 903 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 903 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 903 has memoization hash 21d2e71d5619128a479bd44dade809cb parsl.dataflow.memoization INFO: Task 903 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529664360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 903 try 0 launched on executor multi with executor id 302 parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/d6fcb666-0a91-4534-8bd5-c5e13da00de7_isr_1124707_1.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/d6fcb666-0a91-4534-8bd5-c5e13da00de7_isr_1124707_1.stderr parsl.dataflow.dflow DEBUG: Task 904 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 904 submitted for App characterizeImage, waiting on task 903 parsl.dataflow.dflow DEBUG: Task 904 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 904 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 905 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 905 submitted for App calibrate, waiting on task 904 parsl.dataflow.dflow DEBUG: Task 905 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 905 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 906 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 906 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 906 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 906 has memoization hash bf62743aef00a4879a0e24df5f13ca5c parsl.dataflow.memoization INFO: Task 906 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529667600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 906 try 0 launched on executor multi with executor id 303 parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/f4466f56-8e1e-4333-b78b-0711f119b5f4_isr_1124707_50.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/f4466f56-8e1e-4333-b78b-0711f119b5f4_isr_1124707_50.stderr parsl.dataflow.dflow DEBUG: Task 907 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 907 submitted for App characterizeImage, waiting on task 906 parsl.dataflow.dflow DEBUG: Task 907 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 907 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 908 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 908 submitted for App calibrate, waiting on task 907 parsl.dataflow.dflow DEBUG: Task 908 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 908 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 909 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 909 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 909 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 909 has memoization hash 144bc66aed8ad9364a2893c20528ad36 parsl.dataflow.memoization INFO: Task 909 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296672e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 909 try 0 launched on executor multi with executor id 304 parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/bea03b83-2902-4677-bdd9-8392fcd153e2_isr_1124710_61.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/bea03b83-2902-4677-bdd9-8392fcd153e2_isr_1124710_61.stderr parsl.dataflow.dflow DEBUG: Task 910 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 910 submitted for App characterizeImage, waiting on task 909 parsl.dataflow.dflow DEBUG: Task 910 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 910 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 911 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 911 submitted for App calibrate, waiting on task 910 parsl.dataflow.dflow DEBUG: Task 911 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 911 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 912 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 912 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 912 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 912 has memoization hash ce1be39c21bf350e349716ebfc7d6dae parsl.dataflow.memoization INFO: Task 912 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296647c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 912 try 0 launched on executor multi with executor id 305 parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/498b6c24-b505-4ee9-b29d-8b03a07ba074_isr_1124710_57.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/498b6c24-b505-4ee9-b29d-8b03a07ba074_isr_1124710_57.stderr parsl.dataflow.dflow DEBUG: Task 913 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 913 submitted for App characterizeImage, waiting on task 912 parsl.dataflow.dflow DEBUG: Task 913 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 913 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 914 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 914 submitted for App calibrate, waiting on task 913 parsl.dataflow.dflow DEBUG: Task 914 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 914 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 915 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 915 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 915 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 915 has memoization hash 44ec87e3639357961f3bc9b0339dadc3 parsl.dataflow.memoization INFO: Task 915 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529664a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 915 try 0 launched on executor multi with executor id 306 parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/c0203007-7011-4175-a3ca-c9ce219bebff_isr_1124828_24.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/c0203007-7011-4175-a3ca-c9ce219bebff_isr_1124828_24.stderr parsl.dataflow.dflow DEBUG: Task 916 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 916 submitted for App characterizeImage, waiting on task 915 parsl.dataflow.dflow DEBUG: Task 916 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 916 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 917 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 917 submitted for App calibrate, waiting on task 916 parsl.dataflow.dflow DEBUG: Task 917 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 917 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 918 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 918 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 918 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 918 has memoization hash 897dc8e52ad7c37512d7d3d1f42567ed parsl.dataflow.memoization INFO: Task 918 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529664d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 918 try 0 launched on executor multi with executor id 307 parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/fa85e704-49b8-4af3-b102-b51eb508080c_isr_1124704_48.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/fa85e704-49b8-4af3-b102-b51eb508080c_isr_1124704_48.stderr parsl.dataflow.dflow DEBUG: Task 919 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 919 submitted for App characterizeImage, waiting on task 918 parsl.dataflow.dflow DEBUG: Task 919 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 919 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 920 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 920 submitted for App calibrate, waiting on task 919 parsl.dataflow.dflow DEBUG: Task 920 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 920 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 921 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 921 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 921 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 921 has memoization hash c27fd324fcb5da0b939d503f1eb48a4a parsl.dataflow.memoization INFO: Task 921 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529665120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 921 try 0 launched on executor multi with executor id 308 parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7ea8b424-1e67-4a98-819a-567ea491fb08_isr_1124831_48.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7ea8b424-1e67-4a98-819a-567ea491fb08_isr_1124831_48.stderr parsl.dataflow.dflow DEBUG: Task 922 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 922 submitted for App characterizeImage, waiting on task 921 parsl.dataflow.dflow DEBUG: Task 922 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 922 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 923 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 923 submitted for App calibrate, waiting on task 922 parsl.dataflow.dflow DEBUG: Task 923 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 923 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 924 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 924 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 924 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 924 has memoization hash 6aedc28371ec063e84e7b157e2005efb parsl.dataflow.memoization INFO: Task 924 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529665440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 924 try 0 launched on executor multi with executor id 309 parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/3a115b33-73b6-44a2-b974-1baf1f89b2cb_isr_1124828_7.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/3a115b33-73b6-44a2-b974-1baf1f89b2cb_isr_1124828_7.stderr parsl.dataflow.dflow DEBUG: Task 925 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 925 submitted for App characterizeImage, waiting on task 924 parsl.dataflow.dflow DEBUG: Task 925 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 925 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 926 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 926 submitted for App calibrate, waiting on task 925 parsl.dataflow.dflow DEBUG: Task 926 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 926 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 927 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 927 submitted for App isr, not waiting on any dependency parsl.dataflow.dflow DEBUG: Task 927 set to pending state with AppFuture: parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 927 has memoization hash 38870b2a1769fcabd7c5f5b4b0caeacf parsl.dataflow.memoization INFO: Task 927 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529665760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 927 try 0 launched on executor multi with executor id 310 parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/8e66cb82-8ec5-4efe-b084-f47e7b675fdc_isr_1124704_36.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/8e66cb82-8ec5-4efe-b084-f47e7b675fdc_isr_1124704_36.stderr parsl.dataflow.dflow DEBUG: Task 928 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 928 submitted for App characterizeImage, waiting on task 927 parsl.dataflow.dflow DEBUG: Task 928 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 928 has outstanding dependencies, so launch_if_ready skipping parsl.dataflow.dflow DEBUG: Task 929 will be sent to executor multi parsl.dataflow.dflow DEBUG: Adding output dependencies parsl.dataflow.dflow INFO: Task 929 submitted for App calibrate, waiting on task 928 parsl.dataflow.dflow DEBUG: Task 929 set to pending state with AppFuture: parsl.dataflow.dflow DEBUG: Task 929 has outstanding dependencies, so launch_if_ready skipping 2024-03-07 07:25:43 proc_lsst.multi:170 [INFO] [multi] job_id 16887756 proc_lsst.multi INFO: [multi] job_id 16887756 2024-03-07 07:25:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 07:25:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887756 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887756 parsl.executors.status_handling DEBUG: Launched block 8 on executor multi with job ID 16887756 parsl.executors.status_handling INFO: Allocated block ID 9 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-9 2024-03-07 07:25:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-9 2024-03-07 07:25:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 07:25:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=9 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:43 proc_lsst.multi:170 [INFO] [multi] job_id 16887757 proc_lsst.multi INFO: [multi] job_id 16887757 2024-03-07 07:25:43 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 07:25:43 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:43 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887757 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887757 parsl.executors.status_handling DEBUG: Launched block 9 on executor multi with job ID 16887757 parsl.executors.status_handling INFO: Allocated block ID 10 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-10 2024-03-07 07:25:43 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-10 2024-03-07 07:25:43 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:43 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:43 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 07:25:43 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:43 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=10 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:44 proc_lsst.multi:170 [INFO] [multi] job_id 16887758 proc_lsst.multi INFO: [multi] job_id 16887758 2024-03-07 07:25:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 07:25:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887758 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887758 parsl.executors.status_handling DEBUG: Launched block 10 on executor multi with job ID 16887758 parsl.executors.status_handling INFO: Allocated block ID 11 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-11 2024-03-07 07:25:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-11 2024-03-07 07:25:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 07:25:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=11 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:44 proc_lsst.multi:170 [INFO] [multi] job_id 16887759 proc_lsst.multi INFO: [multi] job_id 16887759 2024-03-07 07:25:44 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 07:25:44 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:44 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887759 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887759 parsl.executors.status_handling DEBUG: Launched block 11 on executor multi with job ID 16887759 parsl.executors.status_handling INFO: Allocated block ID 12 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-12 2024-03-07 07:25:44 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-12 2024-03-07 07:25:44 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:44 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:44 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 07:25:44 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:44 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=12 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:45 proc_lsst.multi:170 [INFO] [multi] job_id 16887760 proc_lsst.multi INFO: [multi] job_id 16887760 2024-03-07 07:25:45 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 07:25:45 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:45 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887760 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887760 parsl.executors.status_handling DEBUG: Launched block 12 on executor multi with job ID 16887760 parsl.executors.status_handling INFO: Allocated block ID 13 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-13 2024-03-07 07:25:45 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-13 2024-03-07 07:25:45 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:45 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:45 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 07:25:45 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:45 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=13 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:47 proc_lsst.multi:170 [INFO] [multi] job_id 16887761 proc_lsst.multi INFO: [multi] job_id 16887761 2024-03-07 07:25:47 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 07:25:47 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:47 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887761 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887761 parsl.executors.status_handling DEBUG: Launched block 13 on executor multi with job ID 16887761 parsl.executors.status_handling INFO: Allocated block ID 14 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-14 2024-03-07 07:25:47 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-14 2024-03-07 07:25:47 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:47 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:47 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 07:25:47 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:47 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=14 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:48 proc_lsst.multi:170 [INFO] [multi] job_id 16887762 proc_lsst.multi INFO: [multi] job_id 16887762 2024-03-07 07:25:48 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 07:25:48 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:48 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887762 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887762 parsl.executors.status_handling DEBUG: Launched block 14 on executor multi with job ID 16887762 parsl.executors.status_handling INFO: Allocated block ID 15 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-15 2024-03-07 07:25:48 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-15 2024-03-07 07:25:48 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:48 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:48 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 07:25:48 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:48 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=15 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:50 proc_lsst.multi:170 [INFO] [multi] job_id 16887763 proc_lsst.multi INFO: [multi] job_id 16887763 2024-03-07 07:25:50 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 07:25:50 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:50 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887763 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887763 parsl.executors.status_handling DEBUG: Launched block 15 on executor multi with job ID 16887763 parsl.executors.status_handling INFO: Allocated block ID 16 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-16 2024-03-07 07:25:50 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-16 2024-03-07 07:25:50 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:50 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:50 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 07:25:50 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:50 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=16 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:51 proc_lsst.multi:170 [INFO] [multi] job_id 16887766 proc_lsst.multi INFO: [multi] job_id 16887766 2024-03-07 07:25:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 07:25:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887766 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887766 parsl.executors.status_handling DEBUG: Launched block 16 on executor multi with job ID 16887766 parsl.executors.status_handling INFO: Allocated block ID 17 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-17 2024-03-07 07:25:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-17 2024-03-07 07:25:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 07:25:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=17 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:51 proc_lsst.multi:170 [INFO] [multi] job_id 16887767 proc_lsst.multi INFO: [multi] job_id 16887767 2024-03-07 07:25:51 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 07:25:51 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:51 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887767 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887767 parsl.executors.status_handling DEBUG: Launched block 17 on executor multi with job ID 16887767 parsl.executors.status_handling INFO: Allocated block ID 18 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-18 2024-03-07 07:25:51 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-18 2024-03-07 07:25:51 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:51 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:51 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 07:25:51 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:51 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=18 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:52 proc_lsst.multi:170 [INFO] [multi] job_id 16887768 proc_lsst.multi INFO: [multi] job_id 16887768 2024-03-07 07:25:52 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 07:25:52 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:52 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887768 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887768 parsl.executors.status_handling DEBUG: Launched block 18 on executor multi with job ID 16887768 parsl.executors.status_handling INFO: Allocated block ID 19 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-19 2024-03-07 07:25:52 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-19 2024-03-07 07:25:52 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:52 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:52 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 07:25:52 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:52 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=19 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:53 proc_lsst.multi:170 [INFO] [multi] job_id 16887769 proc_lsst.multi INFO: [multi] job_id 16887769 2024-03-07 07:25:53 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 07:25:53 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:53 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887769 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887769 parsl.executors.status_handling DEBUG: Launched block 19 on executor multi with job ID 16887769 parsl.executors.status_handling INFO: Allocated block ID 20 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-20 2024-03-07 07:25:53 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-20 2024-03-07 07:25:53 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:53 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:53 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 07:25:53 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:53 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=20 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:54 proc_lsst.multi:170 [INFO] [multi] job_id 16887770 proc_lsst.multi INFO: [multi] job_id 16887770 2024-03-07 07:25:54 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:25:54 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:54 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887770 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887770 parsl.executors.status_handling DEBUG: Launched block 20 on executor multi with job ID 16887770 parsl.executors.status_handling INFO: Allocated block ID 21 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-21 2024-03-07 07:25:54 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-21 2024-03-07 07:25:54 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:54 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:54 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:25:54 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:54 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=21 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:55 proc_lsst.multi:170 [INFO] [multi] job_id 16887771 proc_lsst.multi INFO: [multi] job_id 16887771 2024-03-07 07:25:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 07:25:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887771 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887771 parsl.executors.status_handling DEBUG: Launched block 21 on executor multi with job ID 16887771 parsl.executors.status_handling INFO: Allocated block ID 22 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-22 2024-03-07 07:25:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-22 2024-03-07 07:25:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 07:25:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=22 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:55 proc_lsst.multi:170 [INFO] [multi] job_id 16887772 proc_lsst.multi INFO: [multi] job_id 16887772 2024-03-07 07:25:55 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 07:25:55 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:55 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887772 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887772 parsl.executors.status_handling DEBUG: Launched block 22 on executor multi with job ID 16887772 parsl.executors.status_handling INFO: Allocated block ID 23 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-23 2024-03-07 07:25:55 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-23 2024-03-07 07:25:55 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:55 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:55 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 07:25:55 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:55 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=23 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:56 proc_lsst.multi:170 [INFO] [multi] job_id 16887773 proc_lsst.multi INFO: [multi] job_id 16887773 2024-03-07 07:25:56 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 07:25:56 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:56 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887773 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887773 parsl.executors.status_handling DEBUG: Launched block 23 on executor multi with job ID 16887773 parsl.executors.status_handling INFO: Allocated block ID 24 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-24 2024-03-07 07:25:56 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-24 2024-03-07 07:25:56 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:56 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:56 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 23 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 23 2024-03-07 07:25:56 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:56 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=24 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 0 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bc1feaeac694e25a0ff8a10901da94b4 with result from task 0 parsl.dataflow.dflow DEBUG: Task 0 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 1 has memoization hash 3f434be0ed1ae98f7750bbe444c4bc59 parsl.dataflow.memoization INFO: Task 1 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad48cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 1 try 0 launched on executor multi with executor id 311 parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/4be1f4be-c0ab-42e2-b1b1-2c302ab4d21a_characterizeImage_1124831_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/4be1f4be-c0ab-42e2-b1b1-2c302ab4d21a_characterizeImage_1124831_45.stderr parsl.dataflow.dflow INFO: Standard output for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/be798d22-6cbd-4fcc-a8e3-525cf8e9396e_isr_1124831_45.stdout parsl.dataflow.dflow INFO: Standard error for task 0 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/be798d22-6cbd-4fcc-a8e3-525cf8e9396e_isr_1124831_45.stderr 2024-03-07 07:25:57 proc_lsst.multi:170 [INFO] [multi] job_id 16887774 proc_lsst.multi INFO: [multi] job_id 16887774 2024-03-07 07:25:57 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 07:25:57 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:57 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887774 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887774 parsl.executors.status_handling DEBUG: Launched block 24 on executor multi with job ID 16887774 parsl.executors.status_handling INFO: Allocated block ID 25 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-25 2024-03-07 07:25:57 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-25 2024-03-07 07:25:57 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:57 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:57 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 24 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 24 2024-03-07 07:25:57 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:57 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=25 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:25:58 proc_lsst.multi:170 [INFO] [multi] job_id 16887775 proc_lsst.multi INFO: [multi] job_id 16887775 2024-03-07 07:25:58 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 07:25:58 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:58 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887775 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887775 parsl.executors.status_handling DEBUG: Launched block 25 on executor multi with job ID 16887775 parsl.executors.status_handling INFO: Allocated block ID 26 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-26 2024-03-07 07:25:58 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-26 2024-03-07 07:25:58 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:58 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:58 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 25 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 25 2024-03-07 07:25:58 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:58 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=26 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 6 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef2d0c2919e8e66f0fb8a9f5a688fdbb with result from task 6 parsl.dataflow.dflow DEBUG: Task 6 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 88040c6e7c7cf0d31ccdaf37bb808a6e parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad496c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 0 launched on executor multi with executor id 312 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bdc00ba4-6611-4e7e-9998-eb57597fa382_characterizeImage_1124704_2.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bdc00ba4-6611-4e7e-9998-eb57597fa382_characterizeImage_1124704_2.stderr parsl.dataflow.dflow INFO: Standard output for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/06ab3d96-5d3e-408a-9f33-6234e8a873bc_isr_1124704_2.stdout parsl.dataflow.dflow INFO: Standard error for task 6 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/06ab3d96-5d3e-408a-9f33-6234e8a873bc_isr_1124704_2.stderr 2024-03-07 07:25:59 proc_lsst.multi:170 [INFO] [multi] job_id 16887776 proc_lsst.multi INFO: [multi] job_id 16887776 2024-03-07 07:25:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 07:25:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887776 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887776 parsl.executors.status_handling DEBUG: Launched block 26 on executor multi with job ID 16887776 parsl.executors.status_handling INFO: Allocated block ID 27 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-27 2024-03-07 07:25:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-27 2024-03-07 07:25:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 26 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 26 2024-03-07 07:25:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=27 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 3 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7da6b142842a48fd3ab56a904b7871d6 with result from task 3 parsl.dataflow.dflow DEBUG: Task 3 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 4 has memoization hash 86b8de94091d790cd4287d0f965bd624 parsl.dataflow.memoization INFO: Task 4 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad491c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 4 try 0 launched on executor multi with executor id 313 parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/1679141f-8292-4f09-86fe-adbe1ccff61e_characterizeImage_1124831_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/1679141f-8292-4f09-86fe-adbe1ccff61e_characterizeImage_1124831_3.stderr parsl.dataflow.dflow INFO: Standard output for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/fed63d7b-eb74-4cd5-9884-d90b39b6c920_isr_1124831_3.stdout parsl.dataflow.dflow INFO: Standard error for task 3 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/fed63d7b-eb74-4cd5-9884-d90b39b6c920_isr_1124831_3.stderr parsl.dataflow.dflow INFO: Task 15 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62dd640c64bd61629b0ebf31870d9ab5 with result from task 15 parsl.dataflow.dflow DEBUG: Task 15 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 33bb2dcd21588a19fda78b0824addbe2 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 0 launched on executor multi with executor id 314 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/98428587-0f80-48cf-b5ae-ec3269d4888b_characterizeImage_1124828_2.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/98428587-0f80-48cf-b5ae-ec3269d4888b_characterizeImage_1124828_2.stderr parsl.dataflow.dflow INFO: Standard output for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/75b20162-2ace-466f-904b-dd59f917d480_isr_1124828_2.stdout parsl.dataflow.dflow INFO: Standard error for task 15 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/75b20162-2ace-466f-904b-dd59f917d480_isr_1124828_2.stderr 2024-03-07 07:25:59 proc_lsst.multi:170 [INFO] [multi] job_id 16887777 proc_lsst.multi INFO: [multi] job_id 16887777 2024-03-07 07:25:59 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 07:25:59 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:59 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887777 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887777 parsl.executors.status_handling DEBUG: Launched block 27 on executor multi with job ID 16887777 parsl.executors.status_handling INFO: Allocated block ID 28 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-28 2024-03-07 07:25:59 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-28 2024-03-07 07:25:59 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:25:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:25:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:25:59 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:25:59 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 27 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 27 2024-03-07 07:25:59 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:25:59 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=28 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:00 proc_lsst.multi:170 [INFO] [multi] job_id 16887778 proc_lsst.multi INFO: [multi] job_id 16887778 2024-03-07 07:26:00 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 07:26:00 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:00 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887778 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887778 parsl.executors.status_handling DEBUG: Launched block 28 on executor multi with job ID 16887778 parsl.executors.status_handling INFO: Allocated block ID 29 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-29 2024-03-07 07:26:00 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-29 2024-03-07 07:26:00 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:00 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:00 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 28 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 28 2024-03-07 07:26:00 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:00 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=29 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 9 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a62f3dfb77aad1f9e43eb19cfa71cf8f with result from task 9 parsl.dataflow.dflow DEBUG: Task 9 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 10 has memoization hash 6d745c990f8bab85b09cd0cc4a3f2288 parsl.dataflow.memoization INFO: Task 10 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad499e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 10 try 0 launched on executor multi with executor id 315 parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/e6d1ac57-a151-401d-8b2a-aa6bc3130491_characterizeImage_1124831_5.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/e6d1ac57-a151-401d-8b2a-aa6bc3130491_characterizeImage_1124831_5.stderr parsl.dataflow.dflow INFO: Standard output for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/49552ea2-fcc4-413e-bcd1-ea8d5630860e_isr_1124831_5.stdout parsl.dataflow.dflow INFO: Standard error for task 9 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/49552ea2-fcc4-413e-bcd1-ea8d5630860e_isr_1124831_5.stderr 2024-03-07 07:26:01 proc_lsst.multi:170 [INFO] [multi] job_id 16887779 proc_lsst.multi INFO: [multi] job_id 16887779 2024-03-07 07:26:01 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 07:26:01 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:01 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887779 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887779 parsl.executors.status_handling DEBUG: Launched block 29 on executor multi with job ID 16887779 parsl.executors.status_handling INFO: Allocated block ID 30 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-30 2024-03-07 07:26:01 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-30 2024-03-07 07:26:01 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:01 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:01 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 29 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 29 2024-03-07 07:26:01 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:01 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=30 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to astro parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 21 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 794f7f9ed9b6535800c9dd298e5849c0 with result from task 21 parsl.dataflow.dflow DEBUG: Task 21 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 22 has memoization hash 5db88f2d7071a90b93304a0cd6cb7917 parsl.dataflow.memoization INFO: Task 22 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 22 try 0 launched on executor multi with executor id 316 parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/71cda3a8-370b-44d3-82b5-ac5d628495e5_characterizeImage_1124828_21.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/71cda3a8-370b-44d3-82b5-ac5d628495e5_characterizeImage_1124828_21.stderr parsl.dataflow.dflow INFO: Standard output for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/9707668c-cbdf-474f-b1f1-cb4ff2bd057b_isr_1124828_21.stdout parsl.dataflow.dflow INFO: Standard error for task 21 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/9707668c-cbdf-474f-b1f1-cb4ff2bd057b_isr_1124828_21.stderr parsl.dataflow.dflow INFO: Task 24 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f653dc433b00e6432de5582b5b78d672 with result from task 24 parsl.dataflow.dflow DEBUG: Task 24 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 25 has memoization hash 74ebe4b61b2f700b751b5080a618dcc8 parsl.dataflow.memoization INFO: Task 25 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 07:26:02 proc_lsst.multi:170 [INFO] [multi] job_id 16887780 proc_lsst.multi INFO: [multi] job_id 16887780 2024-03-07 07:26:02 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:02 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:02 proc_lsst.multi:178 [INFO] [multi] provider astro accepted submit and returned 16887780 proc_lsst.multi INFO: [multi] provider astro accepted submit and returned 16887780 parsl.executors.status_handling DEBUG: Launched block 30 on executor multi with job ID 16887780 parsl.executors.status_handling INFO: Allocated block ID 31 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-31 2024-03-07 07:26:02 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-31 2024-03-07 07:26:02 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:02 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:02 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:02 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 07:26:02 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:02 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=31 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 25 try 0 launched on executor multi with executor id 317 parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/385951e3-a59a-4a84-8e1e-d99124dc93c8_characterizeImage_1124828_51.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/385951e3-a59a-4a84-8e1e-d99124dc93c8_characterizeImage_1124828_51.stderr parsl.dataflow.dflow INFO: Standard output for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/190f9bdf-b788-48a8-aefa-a67390fd62d8_isr_1124828_51.stdout parsl.dataflow.dflow INFO: Standard error for task 24 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/190f9bdf-b788-48a8-aefa-a67390fd62d8_isr_1124828_51.stderr parsl.dataflow.dflow INFO: Task 12 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d389ad3f8069be8fd5c3ae80c2bbab38 with result from task 12 parsl.dataflow.dflow DEBUG: Task 12 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 13 has memoization hash b5925da6d2e4797c08efd15eb600b9d0 parsl.dataflow.memoization INFO: Task 13 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad49da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 13 try 0 launched on executor multi with executor id 318 parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0d9ddfbe-5ff2-410e-9bda-d19ea5688c73_characterizeImage_1124828_28.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0d9ddfbe-5ff2-410e-9bda-d19ea5688c73_characterizeImage_1124828_28.stderr parsl.dataflow.dflow INFO: Standard output for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/e3fa8b93-8332-4852-9682-422ab581354a_isr_1124828_28.stdout parsl.dataflow.dflow INFO: Standard error for task 12 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/e3fa8b93-8332-4852-9682-422ab581354a_isr_1124828_28.stderr 2024-03-07 07:26:03 proc_lsst.multi:170 [INFO] [multi] job_id 16887781 proc_lsst.multi INFO: [multi] job_id 16887781 2024-03-07 07:26:03 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:03 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:03 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16887781 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16887781 parsl.executors.status_handling DEBUG: Launched block 31 on executor multi with job ID 16887781 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 32, active_tasks = 310 parsl.dataflow.strategy DEBUG: Executor multi has 310 active tasks, 1/31 running/pending blocks, and 26 connected workers parsl.dataflow.strategy DEBUG: Strategy case 2: slots are overloaded - (slot_ratio = active_slots/active_tasks) < parallelism parsl.dataflow.strategy DEBUG: Strategy case 2b: active_blocks 32 < max_blocks 184 so scaling out parsl.dataflow.strategy DEBUG: Requesting 46 more blocks parsl.executors.status_handling INFO: Scaling out by 46 blocks parsl.executors.status_handling INFO: Allocated block ID 32 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-32 2024-03-07 07:26:03 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-32 2024-03-07 07:26:03 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:03 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:03 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:03 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:03 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:03 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=32 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:04 proc_lsst.multi:170 [INFO] [multi] job_id 16887782 proc_lsst.multi INFO: [multi] job_id 16887782 2024-03-07 07:26:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 07:26:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:04 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16887782 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16887782 parsl.executors.status_handling DEBUG: Launched block 32 on executor multi with job ID 16887782 parsl.executors.status_handling INFO: Allocated block ID 33 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-33 2024-03-07 07:26:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-33 2024-03-07 07:26:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 07:26:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=33 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 18 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e121582dfd2903ece176aeff4354383 with result from task 18 parsl.dataflow.dflow DEBUG: Task 18 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 19 has memoization hash 98af69f772a54546a5cedfad25cc06bf parsl.dataflow.memoization INFO: Task 19 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 19 try 0 launched on executor multi with executor id 319 parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/dcc67948-e9ee-468f-b6f1-31d90d0923e9_characterizeImage_1124710_37.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/dcc67948-e9ee-468f-b6f1-31d90d0923e9_characterizeImage_1124710_37.stderr parsl.dataflow.dflow INFO: Standard output for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f8fd87f4-b505-474b-81da-78b7f150b582_isr_1124710_37.stdout parsl.dataflow.dflow INFO: Standard error for task 18 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f8fd87f4-b505-474b-81da-78b7f150b582_isr_1124710_37.stderr 2024-03-07 07:26:04 proc_lsst.multi:170 [INFO] [multi] job_id 16887783 proc_lsst.multi INFO: [multi] job_id 16887783 2024-03-07 07:26:04 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 07:26:04 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:04 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16887783 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16887783 parsl.executors.status_handling DEBUG: Launched block 33 on executor multi with job ID 16887783 parsl.executors.status_handling INFO: Allocated block ID 34 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-34 2024-03-07 07:26:04 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-34 2024-03-07 07:26:04 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:04 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:04 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:04 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 07:26:04 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:04 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=34 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:05 proc_lsst.multi:170 [INFO] [multi] job_id 16887784 proc_lsst.multi INFO: [multi] job_id 16887784 2024-03-07 07:26:05 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 07:26:05 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:05 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16887784 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16887784 parsl.executors.status_handling DEBUG: Launched block 34 on executor multi with job ID 16887784 parsl.executors.status_handling INFO: Allocated block ID 35 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-35 2024-03-07 07:26:05 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-35 2024-03-07 07:26:05 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:05 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:05 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:05 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 07:26:05 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:05 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=35 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_rtx6k parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:06 proc_lsst.multi:170 [INFO] [multi] job_id 16887785 proc_lsst.multi INFO: [multi] job_id 16887785 2024-03-07 07:26:06 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:06 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:06 proc_lsst.multi:178 [INFO] [multi] provider escience_rtx6k accepted submit and returned 16887785 proc_lsst.multi INFO: [multi] provider escience_rtx6k accepted submit and returned 16887785 parsl.executors.status_handling DEBUG: Launched block 35 on executor multi with job ID 16887785 parsl.executors.status_handling INFO: Allocated block ID 36 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-36 2024-03-07 07:26:06 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-36 2024-03-07 07:26:06 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:06 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:06 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:06 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:06 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 07:26:06 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:06 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=36 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 27 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63322d5bc26b466ad06ccbf967313bd3 with result from task 27 parsl.dataflow.dflow DEBUG: Task 27 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 28 has memoization hash b11caadc7d35a9e9b660e7862e7a5654 parsl.dataflow.memoization INFO: Task 28 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 28 try 0 launched on executor multi with executor id 320 parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e31696bb-a747-43b3-af42-a219d5e5abda_characterizeImage_1124707_49.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e31696bb-a747-43b3-af42-a219d5e5abda_characterizeImage_1124707_49.stderr parsl.dataflow.dflow INFO: Standard output for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/71c8e0b6-fdb6-4f95-a1a8-4fd17f06335f_isr_1124707_49.stdout parsl.dataflow.dflow INFO: Standard error for task 27 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/71c8e0b6-fdb6-4f95-a1a8-4fd17f06335f_isr_1124707_49.stderr 2024-03-07 07:26:07 proc_lsst.multi:170 [INFO] [multi] job_id 16887786 proc_lsst.multi INFO: [multi] job_id 16887786 2024-03-07 07:26:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:07 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887786 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887786 parsl.executors.status_handling DEBUG: Launched block 36 on executor multi with job ID 16887786 parsl.executors.status_handling INFO: Allocated block ID 37 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-37 2024-03-07 07:26:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-37 2024-03-07 07:26:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=37 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:07 proc_lsst.multi:170 [INFO] [multi] job_id 16887787 proc_lsst.multi INFO: [multi] job_id 16887787 2024-03-07 07:26:07 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 07:26:07 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:07 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887787 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887787 parsl.executors.status_handling DEBUG: Launched block 37 on executor multi with job ID 16887787 parsl.executors.status_handling INFO: Allocated block ID 38 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-38 2024-03-07 07:26:07 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-38 2024-03-07 07:26:07 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:07 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:07 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:07 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:07 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 07:26:07 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:07 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=38 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 39 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ae00c1aa3027691bdf98dd30b5a9ccb with result from task 39 parsl.dataflow.dflow DEBUG: Task 39 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 40 has memoization hash 06ee2ed5481ce4c190af73a3a94c430a parsl.dataflow.memoization INFO: Task 40 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 40 try 0 launched on executor multi with executor id 321 parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/17d937da-a8a9-4ccc-8070-ee2f7f2c382f_characterizeImage_1124828_32.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/17d937da-a8a9-4ccc-8070-ee2f7f2c382f_characterizeImage_1124828_32.stderr parsl.dataflow.dflow INFO: Standard output for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/9fe215b5-501b-4963-8de5-ebc04249b4fe_isr_1124828_32.stdout parsl.dataflow.dflow INFO: Standard error for task 39 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/9fe215b5-501b-4963-8de5-ebc04249b4fe_isr_1124828_32.stderr parsl.dataflow.dflow INFO: Task 36 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4f645c0a85e302577f3dbc3b07bdced with result from task 36 parsl.dataflow.dflow DEBUG: Task 36 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 37 has memoization hash abf1dc4e977150ff1ec666dd6469c200 parsl.dataflow.memoization INFO: Task 37 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 07:26:09 proc_lsst.multi:170 [INFO] [multi] job_id 16887788 proc_lsst.multi INFO: [multi] job_id 16887788 2024-03-07 07:26:09 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 07:26:09 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:09 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887788 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887788 parsl.executors.status_handling DEBUG: Launched block 38 on executor multi with job ID 16887788 parsl.executors.status_handling INFO: Allocated block ID 39 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-39 2024-03-07 07:26:09 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-39 2024-03-07 07:26:09 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:09 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:09 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:09 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:09 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 07:26:09 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:09 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=39 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 37 try 0 launched on executor multi with executor id 322 parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e0a3a9c0-179c-4c7c-8e59-d3925bb6057b_characterizeImage_1124707_25.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e0a3a9c0-179c-4c7c-8e59-d3925bb6057b_characterizeImage_1124707_25.stderr parsl.dataflow.dflow INFO: Standard output for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0bf287b9-7b36-4fe0-bea4-fde91691012f_isr_1124707_25.stdout parsl.dataflow.dflow INFO: Standard error for task 36 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0bf287b9-7b36-4fe0-bea4-fde91691012f_isr_1124707_25.stderr parsl.dataflow.dflow INFO: Task 33 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 297e8819f84fe5b72d8e6817ce58c74b with result from task 33 parsl.dataflow.dflow DEBUG: Task 33 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 34 has memoization hash f72090c0fd13cf76aceb002d59222c6d parsl.dataflow.memoization INFO: Task 34 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 34 try 0 launched on executor multi with executor id 323 parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/01ad5621-feac-4d5d-ba7b-d60a2d17e021_characterizeImage_1124710_25.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/01ad5621-feac-4d5d-ba7b-d60a2d17e021_characterizeImage_1124710_25.stderr parsl.dataflow.dflow INFO: Standard output for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/e1230913-bf8f-4100-a371-2b1a0948bfef_isr_1124710_25.stdout parsl.dataflow.dflow INFO: Standard error for task 33 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/e1230913-bf8f-4100-a371-2b1a0948bfef_isr_1124710_25.stderr parsl.dataflow.dflow INFO: Task 30 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab66cc26045422a6c9aa1b1fca2b87c1 with result from task 30 parsl.dataflow.dflow DEBUG: Task 30 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 31 has memoization hash 5b9edca1a26b0bdd30eed7a9d3664564 parsl.dataflow.memoization INFO: Task 31 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 31 try 0 launched on executor multi with executor id 324 parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b43c61cd-8889-4302-ac9a-90553ff45d68_characterizeImage_1124710_16.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b43c61cd-8889-4302-ac9a-90553ff45d68_characterizeImage_1124710_16.stderr parsl.dataflow.dflow INFO: Standard output for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/09762abf-0aeb-4d01-ae6e-441eb698f01c_isr_1124710_16.stdout parsl.dataflow.dflow INFO: Standard error for task 30 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/09762abf-0aeb-4d01-ae6e-441eb698f01c_isr_1124710_16.stderr 2024-03-07 07:26:10 proc_lsst.multi:170 [INFO] [multi] job_id 16887789 proc_lsst.multi INFO: [multi] job_id 16887789 2024-03-07 07:26:10 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 07:26:10 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:10 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887789 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887789 parsl.executors.status_handling DEBUG: Launched block 39 on executor multi with job ID 16887789 parsl.executors.status_handling INFO: Allocated block ID 40 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-40 2024-03-07 07:26:10 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-40 2024-03-07 07:26:10 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:10 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:10 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:10 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:10 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 07:26:10 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:10 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=40 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:11 proc_lsst.multi:170 [INFO] [multi] job_id 16887790 proc_lsst.multi INFO: [multi] job_id 16887790 2024-03-07 07:26:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887790 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887790 parsl.executors.status_handling DEBUG: Launched block 40 on executor multi with job ID 16887790 parsl.executors.status_handling INFO: Allocated block ID 41 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-41 2024-03-07 07:26:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-41 2024-03-07 07:26:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=41 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 45 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a89c09619461a3bdbde80380e0459f88 with result from task 45 parsl.dataflow.dflow DEBUG: Task 45 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 46 has memoization hash 99b96d27c484ad42f0277a342757ab20 parsl.dataflow.memoization INFO: Task 46 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 46 try 0 launched on executor multi with executor id 325 parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/724b3603-462f-47fa-8a08-8976c27db3d4_characterizeImage_1124831_33.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/724b3603-462f-47fa-8a08-8976c27db3d4_characterizeImage_1124831_33.stderr parsl.dataflow.dflow INFO: Standard output for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/bd1198d9-6de0-4c50-b46e-3fc790653593_isr_1124831_33.stdout parsl.dataflow.dflow INFO: Standard error for task 45 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/bd1198d9-6de0-4c50-b46e-3fc790653593_isr_1124831_33.stderr 2024-03-07 07:26:11 proc_lsst.multi:170 [INFO] [multi] job_id 16887791 proc_lsst.multi INFO: [multi] job_id 16887791 2024-03-07 07:26:11 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 07:26:11 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:11 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887791 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887791 parsl.executors.status_handling DEBUG: Launched block 41 on executor multi with job ID 16887791 parsl.executors.status_handling INFO: Allocated block ID 42 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-42 2024-03-07 07:26:11 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-42 2024-03-07 07:26:11 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:11 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:11 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:11 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:11 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 07:26:11 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:11 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=42 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 42 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 58b50830b2b8da797f17f9f658440454 with result from task 42 parsl.dataflow.dflow DEBUG: Task 42 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 43 has memoization hash b615df8b6724cdbfcdcfbb21d0b153c0 parsl.dataflow.memoization INFO: Task 43 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 43 try 0 launched on executor multi with executor id 326 parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8058a4c5-eee5-4ebb-9b01-a81dee60eb4c_characterizeImage_1124707_20.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8058a4c5-eee5-4ebb-9b01-a81dee60eb4c_characterizeImage_1124707_20.stderr parsl.dataflow.dflow INFO: Standard output for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/40ceabf1-a82e-4c7f-af40-68837de7c439_isr_1124707_20.stdout parsl.dataflow.dflow INFO: Standard error for task 42 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/40ceabf1-a82e-4c7f-af40-68837de7c439_isr_1124707_20.stderr 2024-03-07 07:26:12 proc_lsst.multi:170 [INFO] [multi] job_id 16887792 proc_lsst.multi INFO: [multi] job_id 16887792 2024-03-07 07:26:12 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 07:26:12 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:12 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887792 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887792 parsl.executors.status_handling DEBUG: Launched block 42 on executor multi with job ID 16887792 parsl.executors.status_handling INFO: Allocated block ID 43 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-43 2024-03-07 07:26:12 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-43 2024-03-07 07:26:12 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:12 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:12 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:12 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:12 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 07:26:12 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:12 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=43 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 48 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 716ce532f28c0e2a357f2c72b9162d44 with result from task 48 parsl.dataflow.dflow DEBUG: Task 48 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 49 has memoization hash e2702d1fe70d043d784024b9b808e699 parsl.dataflow.memoization INFO: Task 49 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) 2024-03-07 07:26:13 proc_lsst.multi:170 [INFO] [multi] job_id 16887793 proc_lsst.multi INFO: [multi] job_id 16887793 2024-03-07 07:26:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 07:26:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887793 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887793 parsl.executors.status_handling DEBUG: Launched block 43 on executor multi with job ID 16887793 parsl.executors.status_handling INFO: Allocated block ID 44 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-44 2024-03-07 07:26:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-44 2024-03-07 07:26:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 07:26:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=44 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Parsl task 49 try 0 launched on executor multi with executor id 327 parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a2cdd4e2-3bed-4115-ab97-582e59662687_characterizeImage_1124828_34.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a2cdd4e2-3bed-4115-ab97-582e59662687_characterizeImage_1124828_34.stderr parsl.dataflow.dflow INFO: Standard output for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/7fa5eda9-3f3e-44c7-a9a3-cc94305469f2_isr_1124828_34.stdout parsl.dataflow.dflow INFO: Standard error for task 48 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/7fa5eda9-3f3e-44c7-a9a3-cc94305469f2_isr_1124828_34.stderr 2024-03-07 07:26:13 proc_lsst.multi:170 [INFO] [multi] job_id 16887794 proc_lsst.multi INFO: [multi] job_id 16887794 2024-03-07 07:26:13 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 07:26:13 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:13 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887794 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887794 parsl.executors.status_handling DEBUG: Launched block 44 on executor multi with job ID 16887794 parsl.executors.status_handling INFO: Allocated block ID 45 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-45 2024-03-07 07:26:13 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-45 2024-03-07 07:26:13 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:13 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:13 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:13 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:13 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 07:26:13 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:13 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=45 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:14 proc_lsst.multi:170 [INFO] [multi] job_id 16887795 proc_lsst.multi INFO: [multi] job_id 16887795 2024-03-07 07:26:14 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 07:26:14 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:14 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887795 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887795 parsl.executors.status_handling DEBUG: Launched block 45 on executor multi with job ID 16887795 parsl.executors.status_handling INFO: Allocated block ID 46 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-46 2024-03-07 07:26:14 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-46 2024-03-07 07:26:14 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:14 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:14 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:14 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:14 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 07:26:14 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:14 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=46 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 60 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 725c7840f1aca20450ecda274a42946c with result from task 60 parsl.dataflow.dflow DEBUG: Task 60 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 61 has memoization hash 5b05359cc98cbdea4ffe6ce5031ac5b9 parsl.dataflow.memoization INFO: Task 61 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad651c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 61 try 0 launched on executor multi with executor id 328 parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/a2424d05-6fbc-42ea-b66f-698bd46e9f52_characterizeImage_1124831_40.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/a2424d05-6fbc-42ea-b66f-698bd46e9f52_characterizeImage_1124831_40.stderr parsl.dataflow.dflow INFO: Standard output for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f1d2229e-8018-4fb0-abfa-f6f7c71c0a9e_isr_1124831_40.stdout parsl.dataflow.dflow INFO: Standard error for task 60 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f1d2229e-8018-4fb0-abfa-f6f7c71c0a9e_isr_1124831_40.stderr parsl.dataflow.dflow INFO: Task 57 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e5ee78ef563a024255feed1348500e43 with result from task 57 parsl.dataflow.dflow DEBUG: Task 57 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 58 has memoization hash 93b1f3b25209fc2e18cb624c3b4c3bd8 parsl.dataflow.memoization INFO: Task 58 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 58 try 0 launched on executor multi with executor id 329 parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/ebc83df7-cfa2-459c-b54d-83579493687c_characterizeImage_1124710_30.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/ebc83df7-cfa2-459c-b54d-83579493687c_characterizeImage_1124710_30.stderr parsl.dataflow.dflow INFO: Standard output for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/10c5a42a-1db1-431f-b032-acfa529b0f5b_isr_1124710_30.stdout parsl.dataflow.dflow INFO: Standard error for task 57 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/10c5a42a-1db1-431f-b032-acfa529b0f5b_isr_1124710_30.stderr 2024-03-07 07:26:15 proc_lsst.multi:170 [INFO] [multi] job_id 16887796 proc_lsst.multi INFO: [multi] job_id 16887796 2024-03-07 07:26:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 07:26:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887796 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887796 parsl.executors.status_handling DEBUG: Launched block 46 on executor multi with job ID 16887796 parsl.executors.status_handling INFO: Allocated block ID 47 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-47 2024-03-07 07:26:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-47 2024-03-07 07:26:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 07:26:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=47 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 78 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6d94e9fa5c4d96d7f4588bef804f2a8 with result from task 78 parsl.dataflow.dflow DEBUG: Task 78 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 79 has memoization hash 247f6d3ba911693bae7e2ef1514049f8 parsl.dataflow.memoization INFO: Task 79 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad66480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 79 try 0 launched on executor multi with executor id 330 parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d0f9ddb7-7029-445c-a2cf-141e39327e6f_characterizeImage_1124710_18.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d0f9ddb7-7029-445c-a2cf-141e39327e6f_characterizeImage_1124710_18.stderr parsl.dataflow.dflow INFO: Standard output for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/610a69d0-70fe-441d-8543-5dc81d726783_isr_1124710_18.stdout parsl.dataflow.dflow INFO: Standard error for task 78 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/610a69d0-70fe-441d-8543-5dc81d726783_isr_1124710_18.stderr 2024-03-07 07:26:15 proc_lsst.multi:170 [INFO] [multi] job_id 16887797 proc_lsst.multi INFO: [multi] job_id 16887797 2024-03-07 07:26:15 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 07:26:15 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:15 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887797 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887797 parsl.executors.status_handling DEBUG: Launched block 47 on executor multi with job ID 16887797 parsl.executors.status_handling INFO: Allocated block ID 48 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-48 2024-03-07 07:26:15 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-48 2024-03-07 07:26:15 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:15 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:15 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:15 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:15 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 07:26:15 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:15 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=48 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:16 proc_lsst.multi:170 [INFO] [multi] job_id 16887798 proc_lsst.multi INFO: [multi] job_id 16887798 2024-03-07 07:26:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 07:26:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887798 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887798 parsl.executors.status_handling DEBUG: Launched block 48 on executor multi with job ID 16887798 parsl.executors.status_handling INFO: Allocated block ID 49 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-49 2024-03-07 07:26:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-49 2024-03-07 07:26:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 07:26:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=49 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:16 proc_lsst.multi:170 [INFO] [multi] job_id 16887799 proc_lsst.multi INFO: [multi] job_id 16887799 2024-03-07 07:26:16 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 07:26:16 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:16 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887799 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887799 parsl.executors.status_handling DEBUG: Launched block 49 on executor multi with job ID 16887799 parsl.executors.status_handling INFO: Allocated block ID 50 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-50 2024-03-07 07:26:16 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-50 2024-03-07 07:26:16 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:16 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:16 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:16 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:16 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 07:26:16 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:16 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=50 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 75 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42bddae0a2a3e8032bf5a999a25cac32 with result from task 75 parsl.dataflow.dflow DEBUG: Task 75 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 76 has memoization hash ad04feacbb4f2738e67fdc7788d34f82 parsl.dataflow.memoization INFO: Task 76 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad66160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 76 try 0 launched on executor multi with executor id 331 parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f4827b79-86c5-4aa4-8038-49f41b460da3_characterizeImage_1124831_30.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f4827b79-86c5-4aa4-8038-49f41b460da3_characterizeImage_1124831_30.stderr parsl.dataflow.dflow INFO: Standard output for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/5e6a4de0-dc4f-4820-808a-b8bd6c853f55_isr_1124831_30.stdout parsl.dataflow.dflow INFO: Standard error for task 75 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/5e6a4de0-dc4f-4820-808a-b8bd6c853f55_isr_1124831_30.stderr parsl.dataflow.dflow INFO: Task 84 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 479bdbae7bbbf235f0e2f54432c07bff with result from task 84 parsl.dataflow.dflow DEBUG: Task 84 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 85 has memoization hash a4e0801ffd36ecfa1ab29fef91e08616 parsl.dataflow.memoization INFO: Task 85 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad66ac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 85 try 0 launched on executor multi with executor id 332 parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/611dc2bd-957b-4f10-9cf9-49b0d9f3b2c8_characterizeImage_1124710_55.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/611dc2bd-957b-4f10-9cf9-49b0d9f3b2c8_characterizeImage_1124710_55.stderr parsl.dataflow.dflow INFO: Standard output for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/599ab672-1375-4e4a-bee9-ab2e9e5a3621_isr_1124710_55.stdout parsl.dataflow.dflow INFO: Standard error for task 84 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/599ab672-1375-4e4a-bee9-ab2e9e5a3621_isr_1124710_55.stderr 2024-03-07 07:26:17 proc_lsst.multi:170 [INFO] [multi] job_id 16887800 proc_lsst.multi INFO: [multi] job_id 16887800 2024-03-07 07:26:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 07:26:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:17 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887800 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887800 parsl.executors.status_handling DEBUG: Launched block 50 on executor multi with job ID 16887800 parsl.executors.status_handling INFO: Allocated block ID 51 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-51 2024-03-07 07:26:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-51 2024-03-07 07:26:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 07:26:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=51 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:17 proc_lsst.multi:170 [INFO] [multi] job_id 16887801 proc_lsst.multi INFO: [multi] job_id 16887801 2024-03-07 07:26:17 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 07:26:17 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:17 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887801 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887801 parsl.executors.status_handling DEBUG: Launched block 51 on executor multi with job ID 16887801 parsl.executors.status_handling INFO: Allocated block ID 52 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-52 2024-03-07 07:26:17 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-52 2024-03-07 07:26:17 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:17 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:17 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:17 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:17 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 07:26:17 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:17 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=52 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:18 proc_lsst.multi:170 [INFO] [multi] job_id 16887802 proc_lsst.multi INFO: [multi] job_id 16887802 2024-03-07 07:26:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 07:26:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887802 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887802 parsl.executors.status_handling DEBUG: Launched block 52 on executor multi with job ID 16887802 parsl.executors.status_handling INFO: Allocated block ID 53 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-53 2024-03-07 07:26:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-53 2024-03-07 07:26:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 07:26:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=53 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 51 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91f07a27982fc8ee308716f48c2405c1 with result from task 51 parsl.dataflow.dflow DEBUG: Task 51 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 52 has memoization hash 8ea26259118583174a3ad6ba36465b6f parsl.dataflow.memoization INFO: Task 52 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 52 try 0 launched on executor multi with executor id 333 parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/004688bb-cf8c-4d54-a2ed-ce745ca8961e_characterizeImage_1124707_41.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/004688bb-cf8c-4d54-a2ed-ce745ca8961e_characterizeImage_1124707_41.stderr parsl.dataflow.dflow INFO: Standard output for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/169d9003-00e9-4f1d-b1fa-ca1f17698f63_isr_1124707_41.stdout parsl.dataflow.dflow INFO: Standard error for task 51 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/169d9003-00e9-4f1d-b1fa-ca1f17698f63_isr_1124707_41.stderr 2024-03-07 07:26:18 proc_lsst.multi:170 [INFO] [multi] job_id 16887803 proc_lsst.multi INFO: [multi] job_id 16887803 2024-03-07 07:26:18 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 07:26:18 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:18 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887803 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887803 parsl.executors.status_handling DEBUG: Launched block 53 on executor multi with job ID 16887803 parsl.executors.status_handling INFO: Allocated block ID 54 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-54 2024-03-07 07:26:18 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-54 2024-03-07 07:26:18 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:18 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:18 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:18 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:18 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 07:26:18 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:18 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=54 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 102 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f00caba3b5f69de4e71937a3a97f79a0 with result from task 102 parsl.dataflow.dflow DEBUG: Task 102 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 103 has memoization hash a828042b0e86f29918bc0dfe27d2614b parsl.dataflow.memoization INFO: Task 103 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 103 try 0 launched on executor multi with executor id 334 parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e974e6db-23cd-45cb-84a7-b4908f1a7850_characterizeImage_1124710_22.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e974e6db-23cd-45cb-84a7-b4908f1a7850_characterizeImage_1124710_22.stderr parsl.dataflow.dflow INFO: Standard output for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/ff449df6-c8bb-455f-a9a3-e280d77c856f_isr_1124710_22.stdout parsl.dataflow.dflow INFO: Standard error for task 102 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/ff449df6-c8bb-455f-a9a3-e280d77c856f_isr_1124710_22.stderr parsl.dataflow.dflow INFO: Task 54 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 017cc84768d23a99ae319417377df602 with result from task 54 parsl.dataflow.dflow DEBUG: Task 54 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 55 has memoization hash 9f4da31e456bc99d6d5d72088d66c7c1 parsl.dataflow.memoization INFO: Task 55 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 55 try 0 launched on executor multi with executor id 335 parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/ffd32b93-4043-41ab-8c96-6345a93dee3a_characterizeImage_1124831_6.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/ffd32b93-4043-41ab-8c96-6345a93dee3a_characterizeImage_1124831_6.stderr parsl.dataflow.dflow INFO: Standard output for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/da8fc843-509a-47a0-b875-716810c3931f_isr_1124831_6.stdout parsl.dataflow.dflow INFO: Standard error for task 54 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/da8fc843-509a-47a0-b875-716810c3931f_isr_1124831_6.stderr 2024-03-07 07:26:19 proc_lsst.multi:170 [INFO] [multi] job_id 16887804 proc_lsst.multi INFO: [multi] job_id 16887804 2024-03-07 07:26:19 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 07:26:19 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:19 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887804 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887804 parsl.executors.status_handling DEBUG: Launched block 54 on executor multi with job ID 16887804 parsl.executors.status_handling INFO: Allocated block ID 55 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-55 2024-03-07 07:26:19 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-55 2024-03-07 07:26:19 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:19 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:19 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:19 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:19 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 07:26:19 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:19 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=55 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to escience_a40 parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:20 proc_lsst.multi:170 [INFO] [multi] job_id 16887805 proc_lsst.multi INFO: [multi] job_id 16887805 2024-03-07 07:26:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:20 proc_lsst.multi:178 [INFO] [multi] provider escience_a40 accepted submit and returned 16887805 proc_lsst.multi INFO: [multi] provider escience_a40 accepted submit and returned 16887805 parsl.executors.status_handling DEBUG: Launched block 55 on executor multi with job ID 16887805 parsl.executors.status_handling INFO: Allocated block ID 56 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-56 2024-03-07 07:26:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-56 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 0 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 0 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=56 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:20 proc_lsst.multi:170 [INFO] [multi] job_id 16887806 proc_lsst.multi INFO: [multi] job_id 16887806 2024-03-07 07:26:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887806 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887806 parsl.executors.status_handling DEBUG: Launched block 56 on executor multi with job ID 16887806 parsl.executors.status_handling INFO: Allocated block ID 57 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-57 2024-03-07 07:26:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-57 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=57 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:20 proc_lsst.multi:170 [INFO] [multi] job_id 16887807 proc_lsst.multi INFO: [multi] job_id 16887807 2024-03-07 07:26:20 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 07:26:20 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:20 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887807 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887807 parsl.executors.status_handling DEBUG: Launched block 57 on executor multi with job ID 16887807 parsl.executors.status_handling INFO: Allocated block ID 58 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-58 2024-03-07 07:26:20 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-58 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:20 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:20 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 2 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 2 2024-03-07 07:26:20 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:20 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=58 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 81 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dec2b2ed52bdb96a3a342b853067ccc8 with result from task 81 parsl.dataflow.dflow DEBUG: Task 81 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 1dbe3e95e947bbde0227f98a31162e9f parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad667a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 0 launched on executor multi with executor id 336 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/7021b40b-e213-4f60-b138-ffdbfbf58281_characterizeImage_1124828_61.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/7021b40b-e213-4f60-b138-ffdbfbf58281_characterizeImage_1124828_61.stderr parsl.dataflow.dflow INFO: Standard output for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/21020276-238d-45ab-9a6a-785b5f99ab25_isr_1124828_61.stdout parsl.dataflow.dflow INFO: Standard error for task 81 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/21020276-238d-45ab-9a6a-785b5f99ab25_isr_1124828_61.stderr 2024-03-07 07:26:23 proc_lsst.multi:170 [INFO] [multi] job_id 16887808 proc_lsst.multi INFO: [multi] job_id 16887808 2024-03-07 07:26:23 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 07:26:23 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:23 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887808 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887808 parsl.executors.status_handling DEBUG: Launched block 58 on executor multi with job ID 16887808 parsl.executors.status_handling INFO: Allocated block ID 59 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-59 2024-03-07 07:26:23 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-59 2024-03-07 07:26:23 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:23 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:23 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:23 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:23 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:23 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 3 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 3 2024-03-07 07:26:23 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:23 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=59 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 105 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70eda686be7278b839003d320fa33ad3 with result from task 105 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 105 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 106 has memoization hash b840248ee0b7b5637a8cec99dc11ad75 parsl.dataflow.memoization INFO: Task 106 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67ec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 106 try 0 launched on executor multi with executor id 337 parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f78d7bdb-06e2-4e84-ae3d-94a7ca1e5927_characterizeImage_1124710_31.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f78d7bdb-06e2-4e84-ae3d-94a7ca1e5927_characterizeImage_1124710_31.stderr parsl.dataflow.dflow INFO: Standard output for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/97ad3848-dc65-46f8-a45a-e8b1630e68cb_isr_1124710_31.stdout parsl.dataflow.dflow INFO: Standard error for task 105 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/97ad3848-dc65-46f8-a45a-e8b1630e68cb_isr_1124710_31.stderr parsl.dataflow.dflow INFO: Task 87 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a875715f8a3e04c976df800712abf73 with result from task 87 parsl.dataflow.dflow DEBUG: Task 87 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 88 has memoization hash 14a951735d77640424df1a16325ba460 parsl.dataflow.memoization INFO: Task 88 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad66de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 88 try 0 launched on executor multi with executor id 338 parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f373ed55-c87b-4204-a8ed-486f33d8a648_characterizeImage_1124710_41.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f373ed55-c87b-4204-a8ed-486f33d8a648_characterizeImage_1124710_41.stderr parsl.dataflow.dflow INFO: Standard output for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a3500b1b-0adb-4295-a23c-fdaf75aeabbe_isr_1124710_41.stdout parsl.dataflow.dflow INFO: Standard error for task 87 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a3500b1b-0adb-4295-a23c-fdaf75aeabbe_isr_1124710_41.stderr parsl.dataflow.dflow INFO: Task 96 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35cc37e46b5bcbee280af13913b54e31 with result from task 96 parsl.dataflow.dflow DEBUG: Task 96 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 97 has memoization hash 6722f02da540abe148d2fc5e6bcc4757 parsl.dataflow.memoization INFO: Task 97 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 97 try 0 launched on executor multi with executor id 339 parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/2b15b24b-ca99-4e51-a7b1-65e97ade9f6c_characterizeImage_1124704_14.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/2b15b24b-ca99-4e51-a7b1-65e97ade9f6c_characterizeImage_1124704_14.stderr parsl.dataflow.dflow INFO: Standard output for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a3f8f162-f2d8-4565-b902-bbc12da8754e_isr_1124704_14.stdout parsl.dataflow.dflow INFO: Standard error for task 96 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a3f8f162-f2d8-4565-b902-bbc12da8754e_isr_1124704_14.stderr parsl.dataflow.dflow INFO: Task 123 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65806d8abebae3bf2ab0e7977a75dd7f with result from task 123 parsl.dataflow.dflow DEBUG: Task 123 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 124 has memoization hash 3bf0cf53df923392f159bd4cd82bf4ba parsl.dataflow.memoization INFO: Task 124 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 124 try 0 launched on executor multi with executor id 340 parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/ccebdbbb-3d11-404b-862f-619e2428bc10_characterizeImage_1124831_28.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/ccebdbbb-3d11-404b-862f-619e2428bc10_characterizeImage_1124831_28.stderr parsl.dataflow.dflow INFO: Standard output for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/c145c9c4-7028-4407-9354-43f2322a0060_isr_1124831_28.stdout parsl.dataflow.dflow INFO: Standard error for task 123 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/c145c9c4-7028-4407-9354-43f2322a0060_isr_1124831_28.stderr 2024-03-07 07:26:24 proc_lsst.multi:170 [INFO] [multi] job_id 16887809 proc_lsst.multi INFO: [multi] job_id 16887809 2024-03-07 07:26:24 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 07:26:24 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:24 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887809 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887809 parsl.executors.status_handling DEBUG: Launched block 59 on executor multi with job ID 16887809 parsl.executors.status_handling INFO: Allocated block ID 60 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-60 2024-03-07 07:26:24 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-60 2024-03-07 07:26:24 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:24 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:24 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:24 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:24 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:24 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 4 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 4 2024-03-07 07:26:24 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:24 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=60 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 90 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c5fd6a854b03976a3ad7d39e5f71483 with result from task 90 parsl.dataflow.dflow DEBUG: Task 90 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 91 has memoization hash cd1588f1f40fa911a40ef34d99f83235 parsl.dataflow.memoization INFO: Task 91 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 91 try 0 launched on executor multi with executor id 341 parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/fdcb3497-b34a-40a4-b2f4-bae18717b6f6_characterizeImage_1124831_25.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/fdcb3497-b34a-40a4-b2f4-bae18717b6f6_characterizeImage_1124831_25.stderr parsl.dataflow.dflow INFO: Standard output for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/974d1369-ab1a-42ae-ab64-4ec601303b03_isr_1124831_25.stdout parsl.dataflow.dflow INFO: Standard error for task 90 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/974d1369-ab1a-42ae-ab64-4ec601303b03_isr_1124831_25.stderr parsl.dataflow.dflow INFO: Task 93 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3cde5038abff51790b89d7780af9088 with result from task 93 parsl.dataflow.dflow DEBUG: Task 93 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 94 has memoization hash 31e0f2ff183c9f55140b405c11849a71 parsl.dataflow.memoization INFO: Task 94 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 94 try 0 launched on executor multi with executor id 342 parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/39e494c5-8360-41fb-b9e9-fcf12e20f507_characterizeImage_1124831_23.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/39e494c5-8360-41fb-b9e9-fcf12e20f507_characterizeImage_1124831_23.stderr parsl.dataflow.dflow INFO: Standard output for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/86357f3e-8744-402b-a680-b70ca28a3723_isr_1124831_23.stdout parsl.dataflow.dflow INFO: Standard error for task 93 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/86357f3e-8744-402b-a680-b70ca28a3723_isr_1124831_23.stderr parsl.dataflow.dflow INFO: Task 99 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c766e3d4efcadb4e42a50a8959451f4 with result from task 99 parsl.dataflow.dflow DEBUG: Task 99 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 100 has memoization hash de714c66f20a9a4fec89b5cefaf6892c parsl.dataflow.memoization INFO: Task 100 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67a60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 100 try 0 launched on executor multi with executor id 343 parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/bff5f210-e9f3-45ce-b6e6-731209fce737_characterizeImage_1124831_27.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/bff5f210-e9f3-45ce-b6e6-731209fce737_characterizeImage_1124831_27.stderr parsl.dataflow.dflow INFO: Standard output for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/ce9f6d31-a10a-4d39-a64f-3690be8b96aa_isr_1124831_27.stdout parsl.dataflow.dflow INFO: Standard error for task 99 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/ce9f6d31-a10a-4d39-a64f-3690be8b96aa_isr_1124831_27.stderr 2024-03-07 07:26:25 proc_lsst.multi:170 [INFO] [multi] job_id 16887810 proc_lsst.multi INFO: [multi] job_id 16887810 2024-03-07 07:26:25 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:25 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:25 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887810 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887810 parsl.executors.status_handling DEBUG: Launched block 60 on executor multi with job ID 16887810 parsl.executors.status_handling INFO: Allocated block ID 61 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-61 2024-03-07 07:26:25 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-61 2024-03-07 07:26:25 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:25 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:25 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:25 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:25 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:25 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:25 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:25 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=61 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 126 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bf71ad492aab2235c0a6e6848ba93a3 with result from task 126 parsl.dataflow.dflow DEBUG: Task 126 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 127 has memoization hash 557b6d69231c25a3cd860b3078678df5 parsl.dataflow.memoization INFO: Task 127 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 127 try 0 launched on executor multi with executor id 344 parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/9a0f653e-dfdb-4c03-b69f-c50e4c6574fb_characterizeImage_1124828_13.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/9a0f653e-dfdb-4c03-b69f-c50e4c6574fb_characterizeImage_1124828_13.stderr parsl.dataflow.dflow INFO: Standard output for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/6b1717a8-ae3c-48ea-ab68-6f4eb357215f_isr_1124828_13.stdout parsl.dataflow.dflow INFO: Standard error for task 126 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/6b1717a8-ae3c-48ea-ab68-6f4eb357215f_isr_1124828_13.stderr parsl.dataflow.dflow INFO: Task 111 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3b0d131e22a6ef73128259709ab16a7 with result from task 111 parsl.dataflow.dflow DEBUG: Task 111 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 112 has memoization hash 6b1ab2d18e319628740c155dae6c3c5b parsl.dataflow.memoization INFO: Task 112 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 112 try 0 launched on executor multi with executor id 345 parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7b0890ed-0e8a-4308-bb96-c85af93aa523_characterizeImage_1124704_49.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7b0890ed-0e8a-4308-bb96-c85af93aa523_characterizeImage_1124704_49.stderr parsl.dataflow.dflow INFO: Standard output for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ce60ec52-85df-4635-8a3b-e7723a09ced9_isr_1124704_49.stdout parsl.dataflow.dflow INFO: Standard error for task 111 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ce60ec52-85df-4635-8a3b-e7723a09ced9_isr_1124704_49.stderr 2024-03-07 07:26:26 proc_lsst.multi:170 [INFO] [multi] job_id 16887811 proc_lsst.multi INFO: [multi] job_id 16887811 2024-03-07 07:26:26 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 07:26:26 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:26 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887811 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887811 parsl.executors.status_handling DEBUG: Launched block 61 on executor multi with job ID 16887811 parsl.executors.status_handling INFO: Allocated block ID 62 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-62 2024-03-07 07:26:26 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-62 2024-03-07 07:26:26 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:26 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:26 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:26 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:26 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:26 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 6 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 6 2024-03-07 07:26:26 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:26 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=62 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 108 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33e5aaf7429ab1b47db22a978d900a0e with result from task 108 parsl.dataflow.dflow DEBUG: Task 108 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 109 has memoization hash 69fefaa500fdf1caf5c71740baf766e2 parsl.dataflow.memoization INFO: Task 109 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 109 try 0 launched on executor multi with executor id 346 parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/968ea4fb-80eb-4249-9c08-99297ed07fef_characterizeImage_1124831_42.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/968ea4fb-80eb-4249-9c08-99297ed07fef_characterizeImage_1124831_42.stderr parsl.dataflow.dflow INFO: Standard output for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7528194f-fcfe-4070-bc9a-1ddfd23e7d14_isr_1124831_42.stdout parsl.dataflow.dflow INFO: Standard error for task 108 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7528194f-fcfe-4070-bc9a-1ddfd23e7d14_isr_1124831_42.stderr parsl.dataflow.dflow INFO: Task 138 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f13ca967b4723fa13aab29f78087918a with result from task 138 parsl.dataflow.dflow DEBUG: Task 138 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 139 has memoization hash 27d987c86d803bf0263664fe14a20c39 parsl.dataflow.memoization INFO: Task 139 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 139 try 0 launched on executor multi with executor id 347 parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1321d47b-a29d-4aad-bbca-8a26d71dacec_characterizeImage_1124710_33.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1321d47b-a29d-4aad-bbca-8a26d71dacec_characterizeImage_1124710_33.stderr parsl.dataflow.dflow INFO: Standard output for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a973179e-6358-4a1e-a681-986f2af57feb_isr_1124710_33.stdout parsl.dataflow.dflow INFO: Standard error for task 138 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a973179e-6358-4a1e-a681-986f2af57feb_isr_1124710_33.stderr parsl.dataflow.dflow INFO: Task 117 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 186d62b5ef3c41148b97421c317f942f with result from task 117 parsl.dataflow.dflow DEBUG: Task 117 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 118 has memoization hash 60ce5713370ace0630595eda7cc0a70e parsl.dataflow.memoization INFO: Task 118 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 118 try 0 launched on executor multi with executor id 348 parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b73ca140-b3f4-4d52-a554-cfe8d2da317b_characterizeImage_1124710_9.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b73ca140-b3f4-4d52-a554-cfe8d2da317b_characterizeImage_1124710_9.stderr parsl.dataflow.dflow INFO: Standard output for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/e15dffb1-ed6f-45b9-ab94-369c2a1d32b9_isr_1124710_9.stdout parsl.dataflow.dflow INFO: Standard error for task 117 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/e15dffb1-ed6f-45b9-ab94-369c2a1d32b9_isr_1124710_9.stderr parsl.dataflow.dflow INFO: Task 129 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e29f93569faa2bfbcf669cbc8f135db with result from task 129 parsl.dataflow.dflow DEBUG: Task 129 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 130 has memoization hash 71dfd99e663a26bd4d06bf74de83b693 parsl.dataflow.memoization INFO: Task 130 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8d9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 130 try 0 launched on executor multi with executor id 349 parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/35394d1c-cfec-4420-b599-0cc935a7a44b_characterizeImage_1124704_22.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/35394d1c-cfec-4420-b599-0cc935a7a44b_characterizeImage_1124704_22.stderr parsl.dataflow.dflow INFO: Standard output for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/23841145-6556-4fec-8575-70d9b7cf6973_isr_1124704_22.stdout parsl.dataflow.dflow INFO: Standard error for task 129 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/23841145-6556-4fec-8575-70d9b7cf6973_isr_1124704_22.stderr parsl.dataflow.dflow INFO: Task 120 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6830f77b060fae2f5e1d6c11e5b6d049 with result from task 120 parsl.dataflow.dflow DEBUG: Task 120 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 121 has memoization hash a6a216f5b6b8957c897d9ec3c22ec7a4 parsl.dataflow.memoization INFO: Task 121 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 121 try 0 launched on executor multi with executor id 350 parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/27ed34c8-680f-4a4d-b10c-cabebd0c9c6f_characterizeImage_1124704_50.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/27ed34c8-680f-4a4d-b10c-cabebd0c9c6f_characterizeImage_1124704_50.stderr parsl.dataflow.dflow INFO: Standard output for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/80233f77-0ec0-47ea-ba91-481b59196155_isr_1124704_50.stdout parsl.dataflow.dflow INFO: Standard error for task 120 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/80233f77-0ec0-47ea-ba91-481b59196155_isr_1124704_50.stderr 2024-03-07 07:26:27 proc_lsst.multi:170 [INFO] [multi] job_id 16887812 proc_lsst.multi INFO: [multi] job_id 16887812 2024-03-07 07:26:27 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 07:26:27 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:27 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887812 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887812 parsl.executors.status_handling DEBUG: Launched block 62 on executor multi with job ID 16887812 parsl.executors.status_handling INFO: Allocated block ID 63 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-63 2024-03-07 07:26:27 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-63 2024-03-07 07:26:27 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:27 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:27 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:27 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:27 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:27 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 7 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 7 2024-03-07 07:26:27 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:27 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=63 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 141 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bfd5db7a9d0001ff039792868ca8084d with result from task 141 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 141 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 142 has memoization hash 2d36859990c41b78afb02ec636f60d77 parsl.dataflow.memoization INFO: Task 142 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 142 try 0 launched on executor multi with executor id 351 parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d6dfe81a-9aa9-4fd9-84a4-e1a0e7ba49e0_characterizeImage_1124710_32.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d6dfe81a-9aa9-4fd9-84a4-e1a0e7ba49e0_characterizeImage_1124710_32.stderr parsl.dataflow.dflow INFO: Standard output for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/53855d15-ac2c-4b49-a0a7-d2d8a556f6aa_isr_1124710_32.stdout parsl.dataflow.dflow INFO: Standard error for task 141 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/53855d15-ac2c-4b49-a0a7-d2d8a556f6aa_isr_1124710_32.stderr 2024-03-07 07:26:28 proc_lsst.multi:170 [INFO] [multi] job_id 16887813 proc_lsst.multi INFO: [multi] job_id 16887813 2024-03-07 07:26:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 07:26:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887813 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887813 parsl.executors.status_handling DEBUG: Launched block 63 on executor multi with job ID 16887813 parsl.executors.status_handling INFO: Allocated block ID 64 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-64 2024-03-07 07:26:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-64 2024-03-07 07:26:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 8 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 8 2024-03-07 07:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=64 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:28 proc_lsst.multi:170 [INFO] [multi] job_id 16887814 proc_lsst.multi INFO: [multi] job_id 16887814 2024-03-07 07:26:28 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 07:26:28 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:28 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887814 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887814 parsl.executors.status_handling DEBUG: Launched block 64 on executor multi with job ID 16887814 parsl.executors.status_handling INFO: Allocated block ID 65 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-65 2024-03-07 07:26:28 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-65 2024-03-07 07:26:28 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:28 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:28 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:28 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:28 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:28 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 9 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 9 2024-03-07 07:26:28 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:28 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=65 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:29 proc_lsst.multi:170 [INFO] [multi] job_id 16887815 proc_lsst.multi INFO: [multi] job_id 16887815 2024-03-07 07:26:29 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 07:26:29 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:29 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887815 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887815 parsl.executors.status_handling DEBUG: Launched block 65 on executor multi with job ID 16887815 parsl.executors.status_handling INFO: Allocated block ID 66 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-66 2024-03-07 07:26:29 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-66 2024-03-07 07:26:29 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:29 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:29 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:29 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:29 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:29 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 10 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 10 2024-03-07 07:26:29 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:29 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=66 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 63 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e5fc2df59618c2cc66020ed20703b1f with result from task 63 parsl.dataflow.dflow DEBUG: Task 63 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 64 has memoization hash 407685f006aca373a62865a4e84c737e parsl.dataflow.memoization INFO: Task 64 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad654e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 64 try 0 launched on executor multi with executor id 352 parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d39f632f-7c64-4136-bc70-30ffb055370f_characterizeImage_1124704_9.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d39f632f-7c64-4136-bc70-30ffb055370f_characterizeImage_1124704_9.stderr parsl.dataflow.dflow INFO: Standard output for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6325b6a6-20e9-46dd-8607-76dec1ab895e_isr_1124704_9.stdout parsl.dataflow.dflow INFO: Standard error for task 63 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6325b6a6-20e9-46dd-8607-76dec1ab895e_isr_1124704_9.stderr parsl.dataflow.dflow INFO: Task 69 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b900a93eabec86277bbb09fa365f4155 with result from task 69 parsl.dataflow.dflow DEBUG: Task 69 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 70 has memoization hash 5fb7f156f8dcacead2f02ea3139da5e9 parsl.dataflow.memoization INFO: Task 70 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad65b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 70 try 0 launched on executor multi with executor id 353 parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/2a8aebfe-2697-4fe1-8621-6c3552a2bf72_characterizeImage_1124831_47.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/2a8aebfe-2697-4fe1-8621-6c3552a2bf72_characterizeImage_1124831_47.stderr parsl.dataflow.dflow INFO: Standard output for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/5d53417c-92cb-4b45-a959-9353fe490628_isr_1124831_47.stdout parsl.dataflow.dflow INFO: Standard error for task 69 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/5d53417c-92cb-4b45-a959-9353fe490628_isr_1124831_47.stderr parsl.dataflow.dflow INFO: Task 72 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc9107800275a5d92fda10ac72f8b392 with result from task 72 parsl.dataflow.dflow DEBUG: Task 72 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 73 has memoization hash bb6316345b7bd63a9aaa6f96c770ae4d parsl.dataflow.memoization INFO: Task 73 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad65e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 73 try 0 launched on executor multi with executor id 354 parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/c19f084a-edb7-4d4d-8670-7654e3e70ec7_characterizeImage_1124831_58.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/c19f084a-edb7-4d4d-8670-7654e3e70ec7_characterizeImage_1124831_58.stderr parsl.dataflow.dflow INFO: Standard output for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/aa1a792e-8052-4151-9b4e-ece23ef0cdb9_isr_1124831_58.stdout parsl.dataflow.dflow INFO: Standard error for task 72 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/aa1a792e-8052-4151-9b4e-ece23ef0cdb9_isr_1124831_58.stderr parsl.dataflow.dflow INFO: Task 135 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c53aa116692d7d8f33dbe9bd64eee1aa with result from task 135 parsl.dataflow.dflow DEBUG: Task 135 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 136 has memoization hash 5e4195b767d06833ea1d06fea6806fce parsl.dataflow.memoization INFO: Task 136 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 136 try 0 launched on executor multi with executor id 355 parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/930935fe-1e32-4372-ad85-2febbfddf6eb_characterizeImage_1124707_36.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/930935fe-1e32-4372-ad85-2febbfddf6eb_characterizeImage_1124707_36.stderr parsl.dataflow.dflow INFO: Standard output for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/fb3cbdf4-29df-4b9a-a908-1867015d43f1_isr_1124707_36.stdout parsl.dataflow.dflow INFO: Standard error for task 135 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/fb3cbdf4-29df-4b9a-a908-1867015d43f1_isr_1124707_36.stderr 2024-03-07 07:26:30 proc_lsst.multi:170 [INFO] [multi] job_id 16887816 proc_lsst.multi INFO: [multi] job_id 16887816 2024-03-07 07:26:30 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 07:26:30 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:30 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887816 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887816 parsl.executors.status_handling DEBUG: Launched block 66 on executor multi with job ID 16887816 parsl.executors.status_handling INFO: Allocated block ID 67 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-67 2024-03-07 07:26:30 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-67 2024-03-07 07:26:30 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:30 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:30 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:30 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:30 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:30 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 11 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 11 2024-03-07 07:26:30 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:30 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=67 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 144 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 02e5318cf6ee5b3e28f35b7d98cdeb2f with result from task 144 parsl.dataflow.dflow DEBUG: Task 144 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 145 has memoization hash d1328491c5fdddfdf0274eb1a2db2976 parsl.dataflow.memoization INFO: Task 145 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8e980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 145 try 0 launched on executor multi with executor id 356 parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/306514a6-39ec-4d8a-81ec-e48a1a5bb822_characterizeImage_1124710_23.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/306514a6-39ec-4d8a-81ec-e48a1a5bb822_characterizeImage_1124710_23.stderr parsl.dataflow.dflow INFO: Standard output for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1454a56b-9ca4-4f71-be41-c7e769db658b_isr_1124710_23.stdout parsl.dataflow.dflow INFO: Standard error for task 144 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1454a56b-9ca4-4f71-be41-c7e769db658b_isr_1124710_23.stderr parsl.dataflow.dflow INFO: Task 66 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7fc5a91626fcd2fb3effe45fa979b35 with result from task 66 parsl.dataflow.dflow DEBUG: Task 66 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 67 has memoization hash cc6099b4e37a2bb55c7675e3936825af parsl.dataflow.memoization INFO: Task 67 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad65800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 67 try 0 launched on executor multi with executor id 357 parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/dbed9bce-2aa8-44fb-bd40-454657e2ade2_characterizeImage_1124828_59.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/dbed9bce-2aa8-44fb-bd40-454657e2ade2_characterizeImage_1124828_59.stderr parsl.dataflow.dflow INFO: Standard output for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/d3f5f44b-8a94-455d-bf37-f6bc6537ed41_isr_1124828_59.stdout parsl.dataflow.dflow INFO: Standard error for task 66 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/d3f5f44b-8a94-455d-bf37-f6bc6537ed41_isr_1124828_59.stderr parsl.dataflow.dflow INFO: Task 147 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8347172d25fc55a442244ece6e6527a with result from task 147 parsl.dataflow.dflow DEBUG: Task 147 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 148 has memoization hash af7078931362430c357994d32ed0989c parsl.dataflow.memoization INFO: Task 148 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 148 try 0 launched on executor multi with executor id 358 parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/fdf62589-747b-4c4c-86ae-381591f93560_characterizeImage_1124831_21.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/fdf62589-747b-4c4c-86ae-381591f93560_characterizeImage_1124831_21.stderr parsl.dataflow.dflow INFO: Standard output for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/058d57b9-1882-4510-b4b1-a5c42c4e309d_isr_1124831_21.stdout parsl.dataflow.dflow INFO: Standard error for task 147 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/058d57b9-1882-4510-b4b1-a5c42c4e309d_isr_1124831_21.stderr 2024-03-07 07:26:31 proc_lsst.multi:170 [INFO] [multi] job_id 16887817 proc_lsst.multi INFO: [multi] job_id 16887817 2024-03-07 07:26:31 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 07:26:31 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:31 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887817 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887817 parsl.executors.status_handling DEBUG: Launched block 67 on executor multi with job ID 16887817 parsl.executors.status_handling INFO: Allocated block ID 68 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-68 2024-03-07 07:26:31 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-68 2024-03-07 07:26:31 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:31 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:31 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:31 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:31 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:31 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 12 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 12 2024-03-07 07:26:31 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:31 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=68 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 132 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddc4d180463e50a51da9744acad21405 with result from task 132 parsl.dataflow.dflow DEBUG: Task 132 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 133 has memoization hash 1163c22824518dce76827f7f5399057e parsl.dataflow.memoization INFO: Task 133 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 133 try 0 launched on executor multi with executor id 359 parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/02b285b8-ec63-44a8-9f25-8b244634ee82_characterizeImage_1124831_20.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/02b285b8-ec63-44a8-9f25-8b244634ee82_characterizeImage_1124831_20.stderr parsl.dataflow.dflow INFO: Standard output for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/50bdfbcd-5af9-407f-829b-9dd8904c9cdf_isr_1124831_20.stdout parsl.dataflow.dflow INFO: Standard error for task 132 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/50bdfbcd-5af9-407f-829b-9dd8904c9cdf_isr_1124831_20.stderr 2024-03-07 07:26:32 proc_lsst.multi:170 [INFO] [multi] job_id 16887818 proc_lsst.multi INFO: [multi] job_id 16887818 2024-03-07 07:26:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 07:26:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887818 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887818 parsl.executors.status_handling DEBUG: Launched block 68 on executor multi with job ID 16887818 parsl.executors.status_handling INFO: Allocated block ID 69 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-69 2024-03-07 07:26:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-69 2024-03-07 07:26:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 13 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 13 2024-03-07 07:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=69 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 114 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04ce89e007fb87d1c76d22d94d8cac0f with result from task 114 parsl.dataflow.dflow DEBUG: Task 114 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 115 has memoization hash 548c036581850f10fa601df41b430568 parsl.dataflow.memoization INFO: Task 115 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8ca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 115 try 0 launched on executor multi with executor id 360 parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/44a6bb3a-2a7f-4946-8b61-129c7e530646_characterizeImage_1124828_58.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/44a6bb3a-2a7f-4946-8b61-129c7e530646_characterizeImage_1124828_58.stderr parsl.dataflow.dflow INFO: Standard output for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/dddd2040-87c4-4663-81ff-6f96f59dcd69_isr_1124828_58.stdout parsl.dataflow.dflow INFO: Standard error for task 114 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/dddd2040-87c4-4663-81ff-6f96f59dcd69_isr_1124828_58.stderr 2024-03-07 07:26:32 proc_lsst.multi:170 [INFO] [multi] job_id 16887819 proc_lsst.multi INFO: [multi] job_id 16887819 2024-03-07 07:26:32 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 07:26:32 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:32 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887819 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887819 parsl.executors.status_handling DEBUG: Launched block 69 on executor multi with job ID 16887819 parsl.executors.status_handling INFO: Allocated block ID 70 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-70 2024-03-07 07:26:32 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-70 2024-03-07 07:26:32 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:32 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:32 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:32 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:32 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:32 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 14 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 14 2024-03-07 07:26:32 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:32 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=70 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 189 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d26217741493bebf42b1b4bdced43c2 with result from task 189 parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow DEBUG: Task 189 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 190 has memoization hash 5128426c8eec1d936c142d1ff749641a parsl.dataflow.memoization INFO: Task 190 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adad8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 190 try 0 launched on executor multi with executor id 361 parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/bd8f5ea9-1bab-4d98-8530-2466e432e79f_characterizeImage_1124707_27.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/bd8f5ea9-1bab-4d98-8530-2466e432e79f_characterizeImage_1124707_27.stderr parsl.dataflow.dflow INFO: Standard output for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/f77b9c4f-8147-4020-8b41-7ccaea74586b_isr_1124707_27.stdout parsl.dataflow.dflow INFO: Standard error for task 189 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/f77b9c4f-8147-4020-8b41-7ccaea74586b_isr_1124707_27.stderr parsl.dataflow.dflow INFO: Task 156 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e863ca4a6c20783acceaece33363935 with result from task 156 parsl.dataflow.dflow DEBUG: Task 156 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 157 has memoization hash dbcac6eb2eb32764a1f45a1acd6f2031 parsl.dataflow.memoization INFO: Task 157 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 157 try 0 launched on executor multi with executor id 362 parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1b53a577-184f-4c6e-8ac3-4bd1e85233bc_characterizeImage_1124710_56.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1b53a577-184f-4c6e-8ac3-4bd1e85233bc_characterizeImage_1124710_56.stderr parsl.dataflow.dflow INFO: Standard output for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/04d9813c-dcd9-483a-ae59-a6a46388a777_isr_1124710_56.stdout parsl.dataflow.dflow INFO: Standard error for task 156 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/04d9813c-dcd9-483a-ae59-a6a46388a777_isr_1124710_56.stderr parsl.dataflow.dflow INFO: Task 195 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78b394c323a479ebf6408d5eb914e513 with result from task 195 parsl.dataflow.dflow DEBUG: Task 195 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 196 has memoization hash 709c05debb8de0e6d38188cf825b1aa2 parsl.dataflow.memoization INFO: Task 196 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adadee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 196 try 0 launched on executor multi with executor id 363 parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/fce7f302-3289-437e-8b70-69c204d4ed7b_characterizeImage_1124831_34.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/fce7f302-3289-437e-8b70-69c204d4ed7b_characterizeImage_1124831_34.stderr parsl.dataflow.dflow INFO: Standard output for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/0f6809a8-088a-4bb8-9e3e-924bdbf0dbf2_isr_1124831_34.stdout parsl.dataflow.dflow INFO: Standard error for task 195 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/0f6809a8-088a-4bb8-9e3e-924bdbf0dbf2_isr_1124831_34.stderr 2024-03-07 07:26:33 proc_lsst.multi:170 [INFO] [multi] job_id 16887820 proc_lsst.multi INFO: [multi] job_id 16887820 2024-03-07 07:26:33 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 07:26:33 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:33 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887820 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887820 parsl.executors.status_handling DEBUG: Launched block 70 on executor multi with job ID 16887820 parsl.executors.status_handling INFO: Allocated block ID 71 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-71 2024-03-07 07:26:33 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-71 2024-03-07 07:26:33 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:33 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:33 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:33 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:33 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:33 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 15 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 15 2024-03-07 07:26:33 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:33 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=71 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 150 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19a14f5932fd704d18eddf9e00fe4306 with result from task 150 parsl.dataflow.dflow DEBUG: Task 150 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 151 has memoization hash 05f4066d889125d62cba8d48b72de52d parsl.dataflow.memoization INFO: Task 151 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 151 try 0 launched on executor multi with executor id 364 parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/1f17d930-23b9-4627-853c-70a00da4d041_characterizeImage_1124828_26.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/1f17d930-23b9-4627-853c-70a00da4d041_characterizeImage_1124828_26.stderr parsl.dataflow.dflow INFO: Standard output for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bf7b019f-6a22-4402-be34-2ca09248c900_isr_1124828_26.stdout parsl.dataflow.dflow INFO: Standard error for task 150 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bf7b019f-6a22-4402-be34-2ca09248c900_isr_1124828_26.stderr 2024-03-07 07:26:34 proc_lsst.multi:170 [INFO] [multi] job_id 16887821 proc_lsst.multi INFO: [multi] job_id 16887821 2024-03-07 07:26:34 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 07:26:34 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:34 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887821 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887821 parsl.executors.status_handling DEBUG: Launched block 71 on executor multi with job ID 16887821 parsl.executors.status_handling INFO: Allocated block ID 72 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-72 2024-03-07 07:26:34 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-72 2024-03-07 07:26:34 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:34 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:34 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:34 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:34 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:34 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 16 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 16 2024-03-07 07:26:34 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:34 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=72 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 192 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b46afb60d4be08ef1e7ba0dbf3b7a3b0 with result from task 192 parsl.dataflow.dflow DEBUG: Task 192 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 193 has memoization hash 50d83e7c3e4648ad1364c5fedcd9fd68 parsl.dataflow.memoization INFO: Task 193 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adadbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 193 try 0 launched on executor multi with executor id 365 parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/31e02587-d961-4b88-9012-74a5427851af_characterizeImage_1124828_11.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/31e02587-d961-4b88-9012-74a5427851af_characterizeImage_1124828_11.stderr parsl.dataflow.dflow INFO: Standard output for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/385e2540-f2e2-4846-b968-21bc1c018f6b_isr_1124828_11.stdout parsl.dataflow.dflow INFO: Standard error for task 192 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/385e2540-f2e2-4846-b968-21bc1c018f6b_isr_1124828_11.stderr parsl.dataflow.dflow INFO: Task 153 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e92a644d904914066ba5fbbc6999f36 with result from task 153 parsl.dataflow.dflow DEBUG: Task 153 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 154 has memoization hash 17729a766d58041840975c4e52baef54 parsl.dataflow.memoization INFO: Task 154 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8f2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 154 try 0 launched on executor multi with executor id 366 parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/3f0cc665-28bd-44a8-b1b5-b1d2c21152bf_characterizeImage_1124831_59.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/3f0cc665-28bd-44a8-b1b5-b1d2c21152bf_characterizeImage_1124831_59.stderr parsl.dataflow.dflow INFO: Standard output for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f23d3132-a0f2-4c1e-9200-319b290ba01b_isr_1124831_59.stdout parsl.dataflow.dflow INFO: Standard error for task 153 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f23d3132-a0f2-4c1e-9200-319b290ba01b_isr_1124831_59.stderr 2024-03-07 07:26:35 proc_lsst.multi:170 [INFO] [multi] job_id 16887822 proc_lsst.multi INFO: [multi] job_id 16887822 2024-03-07 07:26:35 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 07:26:35 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:35 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887822 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887822 parsl.executors.status_handling DEBUG: Launched block 72 on executor multi with job ID 16887822 parsl.executors.status_handling INFO: Allocated block ID 73 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-73 2024-03-07 07:26:35 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-73 2024-03-07 07:26:35 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:35 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:35 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:35 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:35 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:35 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 17 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 17 2024-03-07 07:26:35 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:35 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=73 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:36 proc_lsst.multi:170 [INFO] [multi] job_id 16887823 proc_lsst.multi INFO: [multi] job_id 16887823 2024-03-07 07:26:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 07:26:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887823 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887823 parsl.executors.status_handling DEBUG: Launched block 73 on executor multi with job ID 16887823 parsl.executors.status_handling INFO: Allocated block ID 74 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-74 2024-03-07 07:26:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-74 2024-03-07 07:26:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 18 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 18 2024-03-07 07:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=74 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:36 proc_lsst.multi:170 [INFO] [multi] job_id 16887824 proc_lsst.multi INFO: [multi] job_id 16887824 2024-03-07 07:26:36 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 07:26:36 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:36 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887824 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887824 parsl.executors.status_handling DEBUG: Launched block 74 on executor multi with job ID 16887824 parsl.executors.status_handling INFO: Allocated block ID 75 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-75 2024-03-07 07:26:36 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-75 2024-03-07 07:26:36 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:36 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:36 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:36 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:36 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:36 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 19 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 19 2024-03-07 07:26:36 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:36 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=75 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 210 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f8b983cd7632ff1b90b82c368a80b3d with result from task 210 parsl.dataflow.dflow DEBUG: Task 210 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 211 has memoization hash 09cceba7ef785e512ab711a01a6576dd parsl.dataflow.memoization INFO: Task 211 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 211 try 0 launched on executor multi with executor id 367 parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/dbd3cbdb-d947-4c57-9b6d-5c7af12fe27c_characterizeImage_1124828_9.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/dbd3cbdb-d947-4c57-9b6d-5c7af12fe27c_characterizeImage_1124828_9.stderr parsl.dataflow.dflow INFO: Standard output for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8f43a090-1c63-4895-91d9-a55a6eff5d08_isr_1124828_9.stdout parsl.dataflow.dflow INFO: Standard error for task 210 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8f43a090-1c63-4895-91d9-a55a6eff5d08_isr_1124828_9.stderr 2024-03-07 07:26:37 proc_lsst.multi:170 [INFO] [multi] job_id 16887825 proc_lsst.multi INFO: [multi] job_id 16887825 2024-03-07 07:26:37 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:37 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:37 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887825 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887825 parsl.executors.status_handling DEBUG: Launched block 75 on executor multi with job ID 16887825 parsl.executors.status_handling INFO: Allocated block ID 76 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-76 2024-03-07 07:26:37 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-76 2024-03-07 07:26:37 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:37 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:37 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:37 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:37 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:37 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:37 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:37 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=76 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.providers.slurm.slurm DEBUG: moving files parsl.dataflow.dflow INFO: Task 267 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry df88afd573a26e3015290ddb059a0f5e with result from task 267 parsl.dataflow.dflow DEBUG: Task 267 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 268 has memoization hash 63af4649a3bd860917824739eac1e6c9 parsl.dataflow.memoization INFO: Task 268 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aaa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 268 try 0 launched on executor multi with executor id 368 parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/ada01af9-6351-4eb9-a3b5-3191672af0be_characterizeImage_1124831_19.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/ada01af9-6351-4eb9-a3b5-3191672af0be_characterizeImage_1124831_19.stderr parsl.dataflow.dflow INFO: Standard output for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/2d6cfbd3-f98e-4060-8ce1-29d8659e8f2f_isr_1124831_19.stdout parsl.dataflow.dflow INFO: Standard error for task 267 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/2d6cfbd3-f98e-4060-8ce1-29d8659e8f2f_isr_1124831_19.stderr 2024-03-07 07:26:38 proc_lsst.multi:170 [INFO] [multi] job_id 16887826 proc_lsst.multi INFO: [multi] job_id 16887826 2024-03-07 07:26:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 07:26:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887826 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887826 parsl.executors.status_handling DEBUG: Launched block 76 on executor multi with job ID 16887826 parsl.executors.status_handling INFO: Allocated block ID 77 parsl.executors.status_handling DEBUG: Submitting to provider with job_name parsl.multi.block-77 2024-03-07 07:26:38 proc_lsst.multi:153 [INFO] [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 proc_lsst.multi INFO: [multi] got submit process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 1 parsl.multi.block-77 2024-03-07 07:26:38 proc_lsst.multi:162 [INFO] [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 proc_lsst.multi INFO: [multi] local process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn --max_workers 2 2024-03-07 07:26:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 1 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 1 2024-03-07 07:26:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 1 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 1 2024-03-07 07:26:38 proc_lsst.multi:162 [INFO] [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] astro process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 30 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 30 2024-03-07 07:26:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 30 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 30 2024-03-07 07:26:38 proc_lsst.multi:162 [INFO] [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_rtx6k process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 5 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 5 2024-03-07 07:26:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 5 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 5 2024-03-07 07:26:38 proc_lsst.multi:162 [INFO] [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] escience_a40 process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 20 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 20 2024-03-07 07:26:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 20 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 20 2024-03-07 07:26:38 proc_lsst.multi:162 [INFO] [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn proc_lsst.multi INFO: [multi] checkpoint process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn 2024-03-07 07:26:38 proc_lsst.multi:163 [INFO] [multi] len(self.providers[provider].resources) 21 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 21 2024-03-07 07:26:38 proc_lsst.multi:164 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:38 proc_lsst.multi:166 [INFO] [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint proc_lsst.multi INFO: [multi] submitting process_worker_pool.py -a 127.0.0.1,10.64.65.9,198.48.92.26,169.254.95.120,10.64.129.9,n3009 -p 0 -c 1.0 -m None --poll 10 --task_port=54122 --result_port=54296 --logdir=/mmfs1/gscratch/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/runinfo/000/multi --block_id=77 --hb_period=30 --hb_threshold=120 --cpu-affinity none --available-accelerators --start-method spawn to checkpoint parsl.providers.slurm.slurm DEBUG: Requesting one block with 1 nodes parsl.providers.slurm.slurm DEBUG: Writing submit script parsl.dataflow.dflow INFO: Task 213 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9791d293404492ed1ec246cfe6a64c44 with result from task 213 parsl.dataflow.dflow DEBUG: Task 213 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 214 has memoization hash 3545fc8e8cbdf3bd5b564afb475b7b4e parsl.dataflow.memoization INFO: Task 214 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaf1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 214 try 0 launched on executor multi with executor id 369 parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/cd089f32-bf18-45ac-8d11-311cc2e26207_characterizeImage_1124704_34.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/cd089f32-bf18-45ac-8d11-311cc2e26207_characterizeImage_1124704_34.stderr parsl.dataflow.dflow INFO: Standard output for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ea008fb5-5de5-4159-8946-34a67d875ed3_isr_1124704_34.stdout parsl.dataflow.dflow INFO: Standard error for task 213 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ea008fb5-5de5-4159-8946-34a67d875ed3_isr_1124704_34.stderr parsl.providers.slurm.slurm DEBUG: moving files 2024-03-07 07:26:38 proc_lsst.multi:170 [INFO] [multi] job_id 16887827 proc_lsst.multi INFO: [multi] job_id 16887827 2024-03-07 07:26:38 proc_lsst.multi:171 [INFO] [multi] len(self.providers[provider].resources) 22 proc_lsst.multi INFO: [multi] len(self.providers[provider].resources) 22 2024-03-07 07:26:38 proc_lsst.multi:172 [INFO] [multi] self.providers[provider].max_blocks 128 proc_lsst.multi INFO: [multi] self.providers[provider].max_blocks 128 2024-03-07 07:26:38 proc_lsst.multi:178 [INFO] [multi] provider checkpoint accepted submit and returned 16887827 proc_lsst.multi INFO: [multi] provider checkpoint accepted submit and returned 16887827 parsl.executors.status_handling DEBUG: Launched block 77 on executor multi with job ID 16887827 parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:26:38 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:26:39 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 222 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54a813c1e8a2df9f4494f4f7124ce820 with result from task 222 parsl.dataflow.dflow DEBUG: Task 222 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 223 has memoization hash 718485514fca17902288277ab4e0a623 parsl.dataflow.memoization INFO: Task 223 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adafb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 223 try 0 launched on executor multi with executor id 370 parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7962c7e0-a434-4559-9a12-4e99cb8ad35a_characterizeImage_1124704_56.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7962c7e0-a434-4559-9a12-4e99cb8ad35a_characterizeImage_1124704_56.stderr parsl.dataflow.dflow INFO: Standard output for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/b6128fa6-3351-42a0-a602-7ee8c85714a8_isr_1124704_56.stdout parsl.dataflow.dflow INFO: Standard error for task 222 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/b6128fa6-3351-42a0-a602-7ee8c85714a8_isr_1124704_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:39 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 216 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6fc3934a14d6a41a6b21f9a0a1e82d73 with result from task 216 parsl.dataflow.dflow DEBUG: Task 216 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 217 has memoization hash 63aa7fcca340e6a64570dd40a3a472ba parsl.dataflow.memoization INFO: Task 217 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaf4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 217 try 0 launched on executor multi with executor id 371 parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/66a0d956-60bd-4c94-b433-79ae576cf6a8_characterizeImage_1124710_10.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/66a0d956-60bd-4c94-b433-79ae576cf6a8_characterizeImage_1124710_10.stderr parsl.dataflow.dflow INFO: Standard output for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1f1b6b3e-459d-4bc3-8f8a-853c3c929bb5_isr_1124710_10.stdout parsl.dataflow.dflow INFO: Standard error for task 216 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1f1b6b3e-459d-4bc3-8f8a-853c3c929bb5_isr_1124710_10.stderr parsl.dataflow.dflow INFO: Task 255 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fd01e782e16118a8ee51a12fc2858b2 with result from task 255 parsl.dataflow.dflow DEBUG: Task 255 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 256 has memoization hash 621a639ffcee3a2cdc40d33e0ccd78ef parsl.dataflow.memoization INFO: Task 256 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 256 try 0 launched on executor multi with executor id 372 parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/582bd3cd-9676-48dc-948e-ecce28bc3dda_characterizeImage_1124707_16.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/582bd3cd-9676-48dc-948e-ecce28bc3dda_characterizeImage_1124707_16.stderr parsl.dataflow.dflow INFO: Standard output for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/443442bb-17fb-4f4c-93ba-ca7682c7ffbc_isr_1124707_16.stdout parsl.dataflow.dflow INFO: Standard error for task 255 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/443442bb-17fb-4f4c-93ba-ca7682c7ffbc_isr_1124707_16.stderr parsl.dataflow.dflow INFO: Task 264 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ccfd45b4c38adf017b6bdefb1c77748 with result from task 264 parsl.dataflow.dflow DEBUG: Task 264 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 265 has memoization hash 0e83fc8634710256ff90dcc5f7ac9a72 parsl.dataflow.memoization INFO: Task 265 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aa700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 265 try 0 launched on executor multi with executor id 373 parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/7228e3be-d514-48e8-b0d0-0e250a381e58_characterizeImage_1124831_18.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/7228e3be-d514-48e8-b0d0-0e250a381e58_characterizeImage_1124831_18.stderr parsl.dataflow.dflow INFO: Standard output for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/21647ca3-fc54-4d4d-84e5-b6f45589f7ca_isr_1124831_18.stdout parsl.dataflow.dflow INFO: Standard error for task 264 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/21647ca3-fc54-4d4d-84e5-b6f45589f7ca_isr_1124831_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:40 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 237 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97bce7cfc537a9279e44688f256ca4f8 with result from task 237 parsl.dataflow.dflow DEBUG: Task 237 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 238 has memoization hash 90a097703cd912fe43b7a01b189e7332 parsl.dataflow.memoization INFO: Task 238 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 238 try 0 launched on executor multi with executor id 374 parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/57414bd4-8170-4c5f-a333-8740a2c0fdc8_characterizeImage_1124710_17.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/57414bd4-8170-4c5f-a333-8740a2c0fdc8_characterizeImage_1124710_17.stderr parsl.dataflow.dflow INFO: Standard output for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f2f57971-bf5d-4efc-9019-34bf927199db_isr_1124710_17.stdout parsl.dataflow.dflow INFO: Standard error for task 237 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f2f57971-bf5d-4efc-9019-34bf927199db_isr_1124710_17.stderr parsl.dataflow.dflow INFO: Task 225 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 613be797957a10f321551ef9693869a7 with result from task 225 parsl.dataflow.dflow DEBUG: Task 225 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 226 has memoization hash 125201d7a8f0ff569a4b33bea3499481 parsl.dataflow.memoization INFO: Task 226 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adafe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 226 try 0 launched on executor multi with executor id 375 parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/00c99fee-79fe-4691-a5a0-a6ca0b62e39b_characterizeImage_1124707_37.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/00c99fee-79fe-4691-a5a0-a6ca0b62e39b_characterizeImage_1124707_37.stderr parsl.dataflow.dflow INFO: Standard output for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/42958e77-fef6-45c1-8485-529dad11d3c3_isr_1124707_37.stdout parsl.dataflow.dflow INFO: Standard error for task 225 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/42958e77-fef6-45c1-8485-529dad11d3c3_isr_1124707_37.stderr parsl.dataflow.dflow INFO: Task 258 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1762e910b656bd5ecfd7d3ff5823d2c5 with result from task 258 parsl.dataflow.dflow DEBUG: Task 258 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 259 has memoization hash 61806bd8c3abe7cba5585cfcc6f892df parsl.dataflow.memoization INFO: Task 259 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aa0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 259 try 0 launched on executor multi with executor id 376 parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/eecb1a2a-c324-4bdc-9980-6237c9de2751_characterizeImage_1124828_27.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/eecb1a2a-c324-4bdc-9980-6237c9de2751_characterizeImage_1124828_27.stderr parsl.dataflow.dflow INFO: Standard output for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/4ba19549-345f-4953-972a-4cc66b2c8784_isr_1124828_27.stdout parsl.dataflow.dflow INFO: Standard error for task 258 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/4ba19549-345f-4953-972a-4cc66b2c8784_isr_1124828_27.stderr parsl.dataflow.dflow INFO: Task 270 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fba6a21a25d497eefe930001ac7f1288 with result from task 270 parsl.dataflow.dflow DEBUG: Task 270 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 271 has memoization hash 483378a11ce5ff9d93913532bde57240 parsl.dataflow.memoization INFO: Task 271 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 271 try 0 launched on executor multi with executor id 377 parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/59f04c55-fc13-48a9-8d5f-c098f1ad20e8_characterizeImage_1124707_47.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/59f04c55-fc13-48a9-8d5f-c098f1ad20e8_characterizeImage_1124707_47.stderr parsl.dataflow.dflow INFO: Standard output for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c61860d4-11cb-422e-b95b-75f4d4c5489e_isr_1124707_47.stdout parsl.dataflow.dflow INFO: Standard error for task 270 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c61860d4-11cb-422e-b95b-75f4d4c5489e_isr_1124707_47.stderr parsl.dataflow.dflow INFO: Task 261 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 294fa9bd9d391628e5979e41a014f017 with result from task 261 parsl.dataflow.dflow DEBUG: Task 261 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 262 has memoization hash e77436a85fe8af2356fdae9b1c0c3c14 parsl.dataflow.memoization INFO: Task 262 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aa3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 262 try 0 launched on executor multi with executor id 378 parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/32bac25f-97c6-4233-a005-57df9735bf0e_characterizeImage_1124707_10.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/32bac25f-97c6-4233-a005-57df9735bf0e_characterizeImage_1124707_10.stderr parsl.dataflow.dflow INFO: Standard output for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/e5776da1-c144-4fda-8df7-4df8b6c1fb2b_isr_1124707_10.stdout parsl.dataflow.dflow INFO: Standard error for task 261 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/e5776da1-c144-4fda-8df7-4df8b6c1fb2b_isr_1124707_10.stderr parsl.dataflow.dflow INFO: Task 231 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 680f7a94a9dd0f92ebac52a7b3e67619 with result from task 231 parsl.dataflow.dflow DEBUG: Task 231 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 232 has memoization hash 61b0193de39b5a4d5e973f3b3dec1bd0 parsl.dataflow.memoization INFO: Task 232 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 232 try 0 launched on executor multi with executor id 379 parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/efb7deec-80ec-4290-9d91-8cfe381fce75_characterizeImage_1124710_6.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/efb7deec-80ec-4290-9d91-8cfe381fce75_characterizeImage_1124710_6.stderr parsl.dataflow.dflow INFO: Standard output for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/54df688e-bbe1-4366-8b58-297140581eb2_isr_1124710_6.stdout parsl.dataflow.dflow INFO: Standard error for task 231 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/54df688e-bbe1-4366-8b58-297140581eb2_isr_1124710_6.stderr parsl.dataflow.dflow INFO: Task 228 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e19748c531dfc6ba77e334f514aa4c5b with result from task 228 parsl.dataflow.dflow DEBUG: Task 228 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 229 has memoization hash 6d1dca2ead926b8f95bcc66700d70478 parsl.dataflow.memoization INFO: Task 229 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 229 try 0 launched on executor multi with executor id 380 parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d8d1c4e2-d524-4e4a-b374-91fa828434de_characterizeImage_1124710_50.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d8d1c4e2-d524-4e4a-b374-91fa828434de_characterizeImage_1124710_50.stderr parsl.dataflow.dflow INFO: Standard output for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/c3a1afdc-cd64-44ae-aaa5-89126aeb0c53_isr_1124710_50.stdout parsl.dataflow.dflow INFO: Standard error for task 228 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/c3a1afdc-cd64-44ae-aaa5-89126aeb0c53_isr_1124710_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:41 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 234 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 04b13973f481d0d8a18b35b92ca04a89 with result from task 234 parsl.dataflow.dflow DEBUG: Task 234 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 235 has memoization hash 01be0a2bdc1fd9724d2f6666bf9ca693 parsl.dataflow.memoization INFO: Task 235 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 235 try 0 launched on executor multi with executor id 381 parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/462d258e-d17d-454d-a9f1-58acac371f50_characterizeImage_1124704_43.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/462d258e-d17d-454d-a9f1-58acac371f50_characterizeImage_1124704_43.stderr parsl.dataflow.dflow INFO: Standard output for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a3231432-8848-40c1-89d5-ac1272c2c3c6_isr_1124704_43.stdout parsl.dataflow.dflow INFO: Standard error for task 234 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a3231432-8848-40c1-89d5-ac1272c2c3c6_isr_1124704_43.stderr parsl.dataflow.dflow INFO: Task 273 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48e77d1784f1885b891c312700c96410 with result from task 273 parsl.dataflow.dflow DEBUG: Task 273 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 274 has memoization hash a07af5abc9a8614cab8f192528f216e0 parsl.dataflow.memoization INFO: Task 274 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ab060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 274 try 0 launched on executor multi with executor id 382 parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/02a2948e-15b7-4a07-a988-0ef015106b86_characterizeImage_1124707_3.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/02a2948e-15b7-4a07-a988-0ef015106b86_characterizeImage_1124707_3.stderr parsl.dataflow.dflow INFO: Standard output for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5f47a431-def2-42f3-880f-813b5d7b729d_isr_1124707_3.stdout parsl.dataflow.dflow INFO: Standard error for task 273 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5f47a431-def2-42f3-880f-813b5d7b729d_isr_1124707_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:42 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 240 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80cc014ecc47a6e567c02fe07170571b with result from task 240 parsl.dataflow.dflow DEBUG: Task 240 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 241 has memoization hash f70fea85bce9c6382623d847a747ee75 parsl.dataflow.memoization INFO: Task 241 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 241 try 0 launched on executor multi with executor id 383 parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/5e752f47-83ec-4f69-aa8b-6bb7470b0da1_characterizeImage_1124704_8.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/5e752f47-83ec-4f69-aa8b-6bb7470b0da1_characterizeImage_1124704_8.stderr parsl.dataflow.dflow INFO: Standard output for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/55858837-002b-422c-9169-fee84377d889_isr_1124704_8.stdout parsl.dataflow.dflow INFO: Standard error for task 240 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/55858837-002b-422c-9169-fee84377d889_isr_1124704_8.stderr parsl.dataflow.dflow INFO: Task 183 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfa178d55e9579968e46822f2823881d with result from task 183 parsl.dataflow.dflow DEBUG: Task 183 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 184 has memoization hash c6baa4d79cb28f464de4e28ff7f8f6d4 parsl.dataflow.memoization INFO: Task 184 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adad260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 184 try 0 launched on executor multi with executor id 384 parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/13e8112e-c442-4968-8b3a-19d18f8531c0_characterizeImage_1124828_41.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/13e8112e-c442-4968-8b3a-19d18f8531c0_characterizeImage_1124828_41.stderr parsl.dataflow.dflow INFO: Standard output for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/b660c004-66d2-456d-926c-20b543acce08_isr_1124828_41.stdout parsl.dataflow.dflow INFO: Standard error for task 183 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/b660c004-66d2-456d-926c-20b543acce08_isr_1124828_41.stderr parsl.dataflow.dflow INFO: Task 186 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85e69c4ca71fb48d2378601de322ba4c with result from task 186 parsl.dataflow.dflow DEBUG: Task 186 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 187 has memoization hash 42855f71ca9dddedb3c0e4386a2865e4 parsl.dataflow.memoization INFO: Task 187 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adad580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 187 try 0 launched on executor multi with executor id 385 parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a19296e8-3699-4a35-8cad-14c87a7b38c9_characterizeImage_1124828_42.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a19296e8-3699-4a35-8cad-14c87a7b38c9_characterizeImage_1124828_42.stderr parsl.dataflow.dflow INFO: Standard output for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/b583b7ec-4780-46a6-b61d-1224d3cfd441_isr_1124828_42.stdout parsl.dataflow.dflow INFO: Standard error for task 186 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/b583b7ec-4780-46a6-b61d-1224d3cfd441_isr_1124828_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:43 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 243 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b81884c27c747d20181f3f7e776cec23 with result from task 243 parsl.dataflow.dflow DEBUG: Task 243 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 244 has memoization hash 38ad56c0641f9c747024ac2cc09a738b parsl.dataflow.memoization INFO: Task 244 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 244 try 0 launched on executor multi with executor id 386 parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/db1da635-c7b6-464c-98b2-1754d7da7e5e_characterizeImage_1124704_27.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/db1da635-c7b6-464c-98b2-1754d7da7e5e_characterizeImage_1124704_27.stderr parsl.dataflow.dflow INFO: Standard output for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/849faf6b-be4b-442b-a95b-41cae797e094_isr_1124704_27.stdout parsl.dataflow.dflow INFO: Standard error for task 243 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/849faf6b-be4b-442b-a95b-41cae797e094_isr_1124704_27.stderr parsl.dataflow.dflow INFO: Task 252 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 090ebe02450242fe4dcc83a5f9204229 with result from task 252 parsl.dataflow.dflow DEBUG: Task 252 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 253 has memoization hash 5fb11575da7b75e7c31cac52f3dc26b1 parsl.dataflow.memoization INFO: Task 253 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 253 try 0 launched on executor multi with executor id 387 parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/ab44db52-14ac-4081-afb6-8174c7a2a15b_characterizeImage_1124704_18.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/ab44db52-14ac-4081-afb6-8174c7a2a15b_characterizeImage_1124704_18.stderr parsl.dataflow.dflow INFO: Standard output for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/fb692671-4542-46e7-ab23-665b8958bab1_isr_1124704_18.stdout parsl.dataflow.dflow INFO: Standard error for task 252 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/fb692671-4542-46e7-ab23-665b8958bab1_isr_1124704_18.stderr parsl.dataflow.dflow INFO: Task 246 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 309cd6a69e83c9826966b604c511d061 with result from task 246 parsl.dataflow.dflow DEBUG: Task 246 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 247 has memoization hash c1c2fffbc4a99036423359f5e73ace10 parsl.dataflow.memoization INFO: Task 247 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 247 try 0 launched on executor multi with executor id 388 parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/0171fa6b-fa0a-460f-9ea0-66edf31be47b_characterizeImage_1124704_40.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/0171fa6b-fa0a-460f-9ea0-66edf31be47b_characterizeImage_1124704_40.stderr parsl.dataflow.dflow INFO: Standard output for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e5db3edc-8048-4dc9-8b93-05d2056c839b_isr_1124704_40.stdout parsl.dataflow.dflow INFO: Standard error for task 246 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e5db3edc-8048-4dc9-8b93-05d2056c839b_isr_1124704_40.stderr parsl.dataflow.dflow INFO: Task 177 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac6bdf82bf5c88423a16f3f01ae5c8f4 with result from task 177 parsl.dataflow.dflow DEBUG: Task 177 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 178 has memoization hash 3480215388c717d66266e0f34214fdda parsl.dataflow.memoization INFO: Task 178 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adacc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 178 try 0 launched on executor multi with executor id 389 parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/a62a1edc-bc4e-4695-99f5-0f7e442f58c3_characterizeImage_1124704_60.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/a62a1edc-bc4e-4695-99f5-0f7e442f58c3_characterizeImage_1124704_60.stderr parsl.dataflow.dflow INFO: Standard output for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/72c9285a-ed6e-4816-be1d-02757b8e68c2_isr_1124704_60.stdout parsl.dataflow.dflow INFO: Standard error for task 177 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/72c9285a-ed6e-4816-be1d-02757b8e68c2_isr_1124704_60.stderr parsl.dataflow.dflow INFO: Task 288 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6069010e8a3cb50dac49840741cd0ef9 with result from task 288 parsl.dataflow.dflow DEBUG: Task 288 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 289 has memoization hash 3788ce2c96b8e2917c24e3dd54720071 parsl.dataflow.memoization INFO: Task 289 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297abe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 289 try 0 launched on executor multi with executor id 390 parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/8cd38eda-2837-4f90-9dfa-d15247aa2c17_characterizeImage_1124710_28.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/8cd38eda-2837-4f90-9dfa-d15247aa2c17_characterizeImage_1124710_28.stderr parsl.dataflow.dflow INFO: Standard output for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/c885063a-9244-4cf8-a7cc-ec8b3f6dc276_isr_1124710_28.stdout parsl.dataflow.dflow INFO: Standard error for task 288 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/c885063a-9244-4cf8-a7cc-ec8b3f6dc276_isr_1124710_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:44 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 249 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3816d5fe3a273cffb9ddf3941bbca696 with result from task 249 parsl.dataflow.dflow DEBUG: Task 249 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 250 has memoization hash ba931c6711c32550048616d6da71803e parsl.dataflow.memoization INFO: Task 250 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 250 try 0 launched on executor multi with executor id 391 parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b2df4c89-fa55-4b55-8892-e56c0f133bd8_characterizeImage_1124710_44.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b2df4c89-fa55-4b55-8892-e56c0f133bd8_characterizeImage_1124710_44.stderr parsl.dataflow.dflow INFO: Standard output for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/04eceea2-8bdd-4224-9ae5-c7f1c02fe902_isr_1124710_44.stdout parsl.dataflow.dflow INFO: Standard error for task 249 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/04eceea2-8bdd-4224-9ae5-c7f1c02fe902_isr_1124710_44.stderr parsl.dataflow.dflow INFO: Task 180 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eed074dbfc5e9ae679a0a75a918e0628 with result from task 180 parsl.dataflow.dflow DEBUG: Task 180 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 181 has memoization hash 6d5efde68fa8dc0a121c022f2f54870a parsl.dataflow.memoization INFO: Task 181 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adacf40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 181 try 0 launched on executor multi with executor id 392 parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/62080edc-8329-48e3-86ef-49935b381e07_characterizeImage_1124704_58.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/62080edc-8329-48e3-86ef-49935b381e07_characterizeImage_1124704_58.stderr parsl.dataflow.dflow INFO: Standard output for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/12a317b3-6154-422c-b0c0-8080c4f25834_isr_1124704_58.stdout parsl.dataflow.dflow INFO: Standard error for task 180 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/12a317b3-6154-422c-b0c0-8080c4f25834_isr_1124704_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:45 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 276 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ec1975617fa443aa50d875c751df3ab4 with result from task 276 parsl.dataflow.dflow DEBUG: Task 276 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 277 has memoization hash 2c6533e2bd8072c8535d0fc90d16c197 parsl.dataflow.memoization INFO: Task 277 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ab380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 277 try 0 launched on executor multi with executor id 393 parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/1ca09767-8beb-451f-b05d-15452299f4e9_characterizeImage_1124707_24.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/1ca09767-8beb-451f-b05d-15452299f4e9_characterizeImage_1124707_24.stderr parsl.dataflow.dflow INFO: Standard output for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5e5abb27-b8e2-42f6-b626-cdc5f999728d_isr_1124707_24.stdout parsl.dataflow.dflow INFO: Standard error for task 276 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5e5abb27-b8e2-42f6-b626-cdc5f999728d_isr_1124707_24.stderr parsl.dataflow.dflow INFO: Task 282 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d9b595957b099d3a7f9167943451f6a with result from task 282 parsl.dataflow.dflow DEBUG: Task 282 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 283 has memoization hash 846227783859fa60a78d6a4b00ab5ac9 parsl.dataflow.memoization INFO: Task 283 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ab9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 283 try 0 launched on executor multi with executor id 394 parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/931e8265-ac0a-4e41-82a0-4e5c9a6a6e1f_characterizeImage_1124707_22.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/931e8265-ac0a-4e41-82a0-4e5c9a6a6e1f_characterizeImage_1124707_22.stderr parsl.dataflow.dflow INFO: Standard output for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/aee68d42-cbd9-4eb8-b7ba-2101a2e25c75_isr_1124707_22.stdout parsl.dataflow.dflow INFO: Standard error for task 282 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/aee68d42-cbd9-4eb8-b7ba-2101a2e25c75_isr_1124707_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:45 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 285 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5faab2b77745f86cf6ec8d18ea0a4f5c with result from task 285 parsl.dataflow.dflow DEBUG: Task 285 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 286 has memoization hash ffd4b71955f6eaba8ba49d8ea3a59c22 parsl.dataflow.memoization INFO: Task 286 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297abce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 286 try 0 launched on executor multi with executor id 395 parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/24d13a3e-f8cc-437b-8deb-16759d06ac5f_characterizeImage_1124828_15.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/24d13a3e-f8cc-437b-8deb-16759d06ac5f_characterizeImage_1124828_15.stderr parsl.dataflow.dflow INFO: Standard output for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/65e832cd-9b4e-4086-a457-39a50581461c_isr_1124828_15.stdout parsl.dataflow.dflow INFO: Standard error for task 285 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/65e832cd-9b4e-4086-a457-39a50581461c_isr_1124828_15.stderr parsl.dataflow.dflow INFO: Task 318 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 590c855f8f5721426e01083764f7d044 with result from task 318 parsl.dataflow.dflow DEBUG: Task 318 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 319 has memoization hash d76112bf5a8875caad5c2f8bac39d7ef parsl.dataflow.memoization INFO: Task 319 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cdf80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 319 try 0 launched on executor multi with executor id 396 parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/b4531e41-c441-4a11-8be8-67b766171bd0_characterizeImage_1124828_18.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/b4531e41-c441-4a11-8be8-67b766171bd0_characterizeImage_1124828_18.stderr parsl.dataflow.dflow INFO: Standard output for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/a8359e7d-1cca-41a9-ac0d-a13299011532_isr_1124828_18.stdout parsl.dataflow.dflow INFO: Standard error for task 318 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/a8359e7d-1cca-41a9-ac0d-a13299011532_isr_1124828_18.stderr parsl.dataflow.dflow INFO: Task 315 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 43e9d6c7a32455429ce108645bda0892 with result from task 315 parsl.dataflow.dflow DEBUG: Task 315 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 316 has memoization hash 251eb9ec6c3e81df29fd7b5976195f14 parsl.dataflow.memoization INFO: Task 316 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cdc60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 316 try 0 launched on executor multi with executor id 397 parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/9f37f68f-152e-42df-bf37-448918cbd08a_characterizeImage_1124831_57.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/9f37f68f-152e-42df-bf37-448918cbd08a_characterizeImage_1124831_57.stderr parsl.dataflow.dflow INFO: Standard output for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/2b1d41c8-d34f-4b3a-9c87-f1d50badb8c0_isr_1124831_57.stdout parsl.dataflow.dflow INFO: Standard error for task 315 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/2b1d41c8-d34f-4b3a-9c87-f1d50badb8c0_isr_1124831_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:46 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 201 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a420f731a3a05e95485740e783e2edb with result from task 201 parsl.dataflow.dflow DEBUG: Task 201 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 202 has memoization hash d94010c3f4976eca683f92b5998828db parsl.dataflow.memoization INFO: Task 202 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adae520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 202 try 0 launched on executor multi with executor id 398 parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/7f884313-ced4-44ab-9eb7-c27add222abe_characterizeImage_1124707_17.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/7f884313-ced4-44ab-9eb7-c27add222abe_characterizeImage_1124707_17.stderr parsl.dataflow.dflow INFO: Standard output for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/2d7f146e-31aa-445e-944d-de77039cfebd_isr_1124707_17.stdout parsl.dataflow.dflow INFO: Standard error for task 201 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/2d7f146e-31aa-445e-944d-de77039cfebd_isr_1124707_17.stderr parsl.dataflow.dflow INFO: Task 204 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 943a4e3c3e932e56290d8e696186056b with result from task 204 parsl.dataflow.dflow DEBUG: Task 204 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 205 has memoization hash 16e5b18a9b57885d8f9deeaf0bb68264 parsl.dataflow.memoization INFO: Task 205 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adae840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 205 try 0 launched on executor multi with executor id 399 parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/3ef2af89-1b9c-4224-9188-73b3633c1c2f_characterizeImage_1124704_28.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/3ef2af89-1b9c-4224-9188-73b3633c1c2f_characterizeImage_1124704_28.stderr parsl.dataflow.dflow INFO: Standard output for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/618e5d82-37ca-490a-8f10-4307b463d15b_isr_1124704_28.stdout parsl.dataflow.dflow INFO: Standard error for task 204 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/618e5d82-37ca-490a-8f10-4307b463d15b_isr_1124704_28.stderr parsl.dataflow.dflow INFO: Task 207 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c0e91f105c6c0c1291e003512769f96 with result from task 207 parsl.dataflow.dflow DEBUG: Task 207 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 208 has memoization hash 49c2953b66edf49ff1b234e1d7056940 parsl.dataflow.memoization INFO: Task 208 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaeb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 208 try 0 launched on executor multi with executor id 400 parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/16595c8b-e7d3-46cf-b52d-1f3e5b90345e_characterizeImage_1124710_46.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/16595c8b-e7d3-46cf-b52d-1f3e5b90345e_characterizeImage_1124710_46.stderr parsl.dataflow.dflow INFO: Standard output for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a7e475c6-aec8-49e3-9555-259ae47f9a8c_isr_1124710_46.stdout parsl.dataflow.dflow INFO: Standard error for task 207 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a7e475c6-aec8-49e3-9555-259ae47f9a8c_isr_1124710_46.stderr parsl.dataflow.dflow INFO: Task 198 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9644eb446dba76858c3d341b2abf959 with result from task 198 parsl.dataflow.dflow DEBUG: Task 198 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 199 has memoization hash bf180cc6fa32d5269e7774965711c85f parsl.dataflow.memoization INFO: Task 199 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adae200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 199 try 0 launched on executor multi with executor id 401 parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/7e7d7f8a-9a40-4172-a067-f1b9071d79c9_characterizeImage_1124831_53.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/7e7d7f8a-9a40-4172-a067-f1b9071d79c9_characterizeImage_1124831_53.stderr parsl.dataflow.dflow INFO: Standard output for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/17d34d2b-0c31-49ed-9ede-e5f93e9f3a49_isr_1124831_53.stdout parsl.dataflow.dflow INFO: Standard error for task 198 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/17d34d2b-0c31-49ed-9ede-e5f93e9f3a49_isr_1124831_53.stderr parsl.dataflow.dflow INFO: Task 219 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a7576cfdc7ba7b59ab00fd9e320410b with result from task 219 parsl.dataflow.dflow DEBUG: Task 219 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 220 has memoization hash 15ba0bec44e03fa2357e6f71e7502d03 parsl.dataflow.memoization INFO: Task 220 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaf7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 220 try 0 launched on executor multi with executor id 402 parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/f24a39bb-0c40-4d9f-97f8-95df8a541eb2_characterizeImage_1124828_17.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/f24a39bb-0c40-4d9f-97f8-95df8a541eb2_characterizeImage_1124828_17.stderr parsl.dataflow.dflow INFO: Standard output for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/a9dc9c9d-370a-492c-9c68-ff0eac30304a_isr_1124828_17.stdout parsl.dataflow.dflow INFO: Standard error for task 219 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/a9dc9c9d-370a-492c-9c68-ff0eac30304a_isr_1124828_17.stderr parsl.dataflow.dflow INFO: Task 168 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fde7a01ff85f009e8b7cfecad454eab with result from task 168 parsl.dataflow.dflow DEBUG: Task 168 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 169 has memoization hash 53f8cd51843947d9fe05b8968f453b70 parsl.dataflow.memoization INFO: Task 169 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adac2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 169 try 0 launched on executor multi with executor id 403 parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f7e04282-b7ea-4b73-a36c-9738bee66f45_characterizeImage_1124704_25.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f7e04282-b7ea-4b73-a36c-9738bee66f45_characterizeImage_1124704_25.stderr parsl.dataflow.dflow INFO: Standard output for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/66367b51-dcf0-40f8-b949-8f46b99d598f_isr_1124704_25.stdout parsl.dataflow.dflow INFO: Standard error for task 168 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/66367b51-dcf0-40f8-b949-8f46b99d598f_isr_1124704_25.stderr parsl.dataflow.dflow INFO: Task 165 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfa0339ef63701b0a55a9c5320dbd18d with result from task 165 parsl.dataflow.dflow DEBUG: Task 165 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 166 has memoization hash ee88682a25caf486b8fc1458b2dff22d parsl.dataflow.memoization INFO: Task 166 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 166 try 0 launched on executor multi with executor id 404 parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7912c118-5d91-4783-b108-13fe11b4eaec_characterizeImage_1124704_6.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7912c118-5d91-4783-b108-13fe11b4eaec_characterizeImage_1124704_6.stderr parsl.dataflow.dflow INFO: Standard output for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/65f386b0-2218-404f-9372-51860ccc3ecd_isr_1124704_6.stdout parsl.dataflow.dflow INFO: Standard error for task 165 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/65f386b0-2218-404f-9372-51860ccc3ecd_isr_1124704_6.stderr parsl.dataflow.dflow INFO: Task 171 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8319a449297333abd4804e1040a2a76 with result from task 171 parsl.dataflow.dflow DEBUG: Task 171 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 172 has memoization hash e0ebc61a4214ff078c5124150d650fb9 parsl.dataflow.memoization INFO: Task 172 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adac5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 172 try 0 launched on executor multi with executor id 405 parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/5e46d950-66ac-4a6d-abe4-137e0d65fa9e_characterizeImage_1124707_30.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/5e46d950-66ac-4a6d-abe4-137e0d65fa9e_characterizeImage_1124707_30.stderr parsl.dataflow.dflow INFO: Standard output for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0d77cac8-968a-40aa-a849-4402fea7b425_isr_1124707_30.stdout parsl.dataflow.dflow INFO: Standard error for task 171 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0d77cac8-968a-40aa-a849-4402fea7b425_isr_1124707_30.stderr parsl.dataflow.dflow INFO: Task 174 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8dc242dedd1e7e528b8d6fdacb5a755c with result from task 174 parsl.dataflow.dflow DEBUG: Task 174 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 175 has memoization hash 0d848e99ab55ec2d9682d5e2a31ab68e parsl.dataflow.memoization INFO: Task 175 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adac900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 175 try 0 launched on executor multi with executor id 406 parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/6c72ebea-cac5-4d8b-ba2d-84dba45c4f64_characterizeImage_1124831_31.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/6c72ebea-cac5-4d8b-ba2d-84dba45c4f64_characterizeImage_1124831_31.stderr parsl.dataflow.dflow INFO: Standard output for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/357f83ce-6317-4bf6-b911-20c3742381d1_isr_1124831_31.stdout parsl.dataflow.dflow INFO: Standard error for task 174 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/357f83ce-6317-4bf6-b911-20c3742381d1_isr_1124831_31.stderr parsl.dataflow.dflow INFO: Task 159 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad2311b92ba0b1ccd1f27191c3a0506d with result from task 159 parsl.dataflow.dflow DEBUG: Task 159 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 160 has memoization hash b50a4462ed1baf61d6c828f93b29c354 parsl.dataflow.memoization INFO: Task 160 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 160 try 0 launched on executor multi with executor id 407 parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/61a9d9c8-e489-414a-9f82-7627149b9f8e_characterizeImage_1124828_62.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/61a9d9c8-e489-414a-9f82-7627149b9f8e_characterizeImage_1124828_62.stderr parsl.dataflow.dflow INFO: Standard output for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/3ceed0e6-cbf7-487c-8e47-c08182d5c563_isr_1124828_62.stdout parsl.dataflow.dflow INFO: Standard error for task 159 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/3ceed0e6-cbf7-487c-8e47-c08182d5c563_isr_1124828_62.stderr parsl.dataflow.dflow INFO: Task 279 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e06f501ee8cf6602af5907b2ae7180e9 with result from task 279 parsl.dataflow.dflow DEBUG: Task 279 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 280 has memoization hash e28ee521b9b21e1f237d4164fd1baf69 parsl.dataflow.memoization INFO: Task 280 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ab6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 280 try 0 launched on executor multi with executor id 408 parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/435de374-89bb-45a6-84c6-c95557499d03_characterizeImage_1124831_52.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/435de374-89bb-45a6-84c6-c95557499d03_characterizeImage_1124831_52.stderr parsl.dataflow.dflow INFO: Standard output for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/8f1fbb28-611c-42ef-9b17-11ebab374ac2_isr_1124831_52.stdout parsl.dataflow.dflow INFO: Standard error for task 279 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/8f1fbb28-611c-42ef-9b17-11ebab374ac2_isr_1124831_52.stderr parsl.dataflow.dflow INFO: Task 162 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ad330097e87aa068b06b3a062c04948 with result from task 162 parsl.dataflow.dflow DEBUG: Task 162 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 163 has memoization hash bd558371c3c59001669b77772899e7e9 parsl.dataflow.memoization INFO: Task 163 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8fc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 163 try 0 launched on executor multi with executor id 409 parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/36eddd68-e58d-40f4-acb2-120a34a6cd65_characterizeImage_1124704_37.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/36eddd68-e58d-40f4-acb2-120a34a6cd65_characterizeImage_1124704_37.stderr parsl.dataflow.dflow INFO: Standard output for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/1b0d04f7-6967-422f-a0fe-1c2a4d82b7e0_isr_1124704_37.stdout parsl.dataflow.dflow INFO: Standard error for task 162 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/1b0d04f7-6967-422f-a0fe-1c2a4d82b7e0_isr_1124704_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:47 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 360 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2f14c129a57bd76ac8627266b55319d9 with result from task 360 parsl.dataflow.dflow DEBUG: Task 360 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 361 has memoization hash 7d80cfd623503c087721e2562c035e84 parsl.dataflow.memoization INFO: Task 361 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 361 try 0 launched on executor multi with executor id 410 parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/f8439488-1eea-451b-bab2-652c930c284c_characterizeImage_1124707_8.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/f8439488-1eea-451b-bab2-652c930c284c_characterizeImage_1124707_8.stderr parsl.dataflow.dflow INFO: Standard output for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/a4185c30-5c0a-4899-b03c-f9b57a8955fd_isr_1124707_8.stdout parsl.dataflow.dflow INFO: Standard error for task 360 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/a4185c30-5c0a-4899-b03c-f9b57a8955fd_isr_1124707_8.stderr parsl.dataflow.dflow INFO: Task 351 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d00b3934069d5b7b3f4409e40d5fd71a with result from task 351 parsl.dataflow.dflow DEBUG: Task 351 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 352 has memoization hash d79cf44d11083d3a459e78f7e7992f2e parsl.dataflow.memoization INFO: Task 352 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 352 try 0 launched on executor multi with executor id 411 parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/efb151e0-9e3e-4ab4-a949-9c5c082e3998_characterizeImage_1124704_4.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/efb151e0-9e3e-4ab4-a949-9c5c082e3998_characterizeImage_1124704_4.stderr parsl.dataflow.dflow INFO: Standard output for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/c2d84418-9f1c-49da-b688-2817a0c11bda_isr_1124704_4.stdout parsl.dataflow.dflow INFO: Standard error for task 351 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/c2d84418-9f1c-49da-b688-2817a0c11bda_isr_1124704_4.stderr parsl.dataflow.dflow INFO: Task 357 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e774147670cd6ee6f0322f6da9cc4274 with result from task 357 parsl.dataflow.dflow DEBUG: Task 357 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 358 has memoization hash 0fa3744afcf3d714d3f33e4a5a347a9f parsl.dataflow.memoization INFO: Task 358 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 358 try 0 launched on executor multi with executor id 412 parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/0218f2db-e2af-4b5f-9ede-1cc1aab1d036_characterizeImage_1124710_43.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/0218f2db-e2af-4b5f-9ede-1cc1aab1d036_characterizeImage_1124710_43.stderr parsl.dataflow.dflow INFO: Standard output for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/2d631d3f-e902-4037-878c-40d7b38dd921_isr_1124710_43.stdout parsl.dataflow.dflow INFO: Standard error for task 357 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/2d631d3f-e902-4037-878c-40d7b38dd921_isr_1124710_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:48 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 363 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae5666296d24ec18d9d57b6539df5de2 with result from task 363 parsl.dataflow.dflow DEBUG: Task 363 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 364 has memoization hash 1c3091b9b049bc466d187dd9a87eecc5 parsl.dataflow.memoization INFO: Task 364 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 364 try 0 launched on executor multi with executor id 413 parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/9dd324a1-8db4-4a04-8db9-9fee074add49_characterizeImage_1124704_19.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/9dd324a1-8db4-4a04-8db9-9fee074add49_characterizeImage_1124704_19.stderr parsl.dataflow.dflow INFO: Standard output for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/9dce3d7c-43ce-45c7-b5b2-5c679efb2c9f_isr_1124704_19.stdout parsl.dataflow.dflow INFO: Standard error for task 363 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/9dce3d7c-43ce-45c7-b5b2-5c679efb2c9f_isr_1124704_19.stderr parsl.dataflow.dflow INFO: Task 354 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46773e63652dfa51c07d74c360e77361 with result from task 354 parsl.dataflow.dflow DEBUG: Task 354 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 355 has memoization hash d3f6b7e227a0e9f5567582c9598235af parsl.dataflow.memoization INFO: Task 355 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 355 try 0 launched on executor multi with executor id 414 parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/fe2a1a13-b65a-4225-a5d7-5ac6fc728611_characterizeImage_1124828_29.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/fe2a1a13-b65a-4225-a5d7-5ac6fc728611_characterizeImage_1124828_29.stderr parsl.dataflow.dflow INFO: Standard output for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/0cb09ee3-3852-4740-94f4-8edec67fa0de_isr_1124828_29.stdout parsl.dataflow.dflow INFO: Standard error for task 354 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/0cb09ee3-3852-4740-94f4-8edec67fa0de_isr_1124828_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:48 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:49 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 348 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98670f06b74e37e12cec5d3f0c27f5f6 with result from task 348 parsl.dataflow.dflow DEBUG: Task 348 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 349 has memoization hash 4b68b342b8838a054aa554a61f3088a6 parsl.dataflow.memoization INFO: Task 349 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cfec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 349 try 0 launched on executor multi with executor id 415 parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/268091b0-1476-41b3-b58e-ed35e3e78544_characterizeImage_1124704_20.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/268091b0-1476-41b3-b58e-ed35e3e78544_characterizeImage_1124704_20.stderr parsl.dataflow.dflow INFO: Standard output for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e2f6b261-c1a1-461e-8c67-26b8085bf9af_isr_1124704_20.stdout parsl.dataflow.dflow INFO: Standard error for task 348 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e2f6b261-c1a1-461e-8c67-26b8085bf9af_isr_1124704_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:50 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 366 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dfc8620607255760ed67f3f0d422a1e with result from task 366 parsl.dataflow.dflow DEBUG: Task 366 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 367 has memoization hash 06b29b672eb8dcffb38d2f26955ac174 parsl.dataflow.memoization INFO: Task 367 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 367 try 0 launched on executor multi with executor id 416 parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/850eb6e1-4341-4588-aabd-e9cbd5d24ce7_characterizeImage_1124710_26.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/850eb6e1-4341-4588-aabd-e9cbd5d24ce7_characterizeImage_1124710_26.stderr parsl.dataflow.dflow INFO: Standard output for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/8029bcb4-6279-419e-b037-19b095ef0e14_isr_1124710_26.stdout parsl.dataflow.dflow INFO: Standard error for task 366 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/8029bcb4-6279-419e-b037-19b095ef0e14_isr_1124710_26.stderr parsl.dataflow.dflow INFO: Task 372 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c83289bb7b69a75a94ea56d1751950fa with result from task 372 parsl.dataflow.dflow DEBUG: Task 372 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 373 has memoization hash 6d291f89802e37d3b24d5dd61ebe0027 parsl.dataflow.memoization INFO: Task 373 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 373 try 0 launched on executor multi with executor id 417 parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/becbcabc-d5eb-4045-adc2-3f6dabc234e4_characterizeImage_1124831_29.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/becbcabc-d5eb-4045-adc2-3f6dabc234e4_characterizeImage_1124831_29.stderr parsl.dataflow.dflow INFO: Standard output for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/9afd980d-28be-422b-858e-31b2734659bc_isr_1124831_29.stdout parsl.dataflow.dflow INFO: Standard error for task 372 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/9afd980d-28be-422b-858e-31b2734659bc_isr_1124831_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:50 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 321 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fe5b1512ea47a73c26abc553e864b3b6 with result from task 321 parsl.dataflow.dflow DEBUG: Task 321 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 322 has memoization hash 14b5083b0774bf7d242c0823a31afcae parsl.dataflow.memoization INFO: Task 322 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ce2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 322 try 0 launched on executor multi with executor id 418 parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/0ed03dd4-3343-4e65-b1b0-ce8e1ae9bef2_characterizeImage_1124831_36.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/0ed03dd4-3343-4e65-b1b0-ce8e1ae9bef2_characterizeImage_1124831_36.stderr parsl.dataflow.dflow INFO: Standard output for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/523d6d35-6cca-4762-9ab3-5a9e160f3790_isr_1124831_36.stdout parsl.dataflow.dflow INFO: Standard error for task 321 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/523d6d35-6cca-4762-9ab3-5a9e160f3790_isr_1124831_36.stderr parsl.dataflow.dflow INFO: Task 300 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b78971599edc05eab4a4044d5c172cea with result from task 300 parsl.dataflow.dflow DEBUG: Task 300 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 301 has memoization hash be866213b1313c6f8f9a1f6d5c559c1b parsl.dataflow.memoization INFO: Task 301 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 301 try 0 launched on executor multi with executor id 419 parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/9eb9a0e1-0ace-4ea9-97c4-17ef03e7dd89_characterizeImage_1124707_43.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/9eb9a0e1-0ace-4ea9-97c4-17ef03e7dd89_characterizeImage_1124707_43.stderr parsl.dataflow.dflow INFO: Standard output for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5868be83-b43f-45b2-b32e-8b3a424ed4e3_isr_1124707_43.stdout parsl.dataflow.dflow INFO: Standard error for task 300 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5868be83-b43f-45b2-b32e-8b3a424ed4e3_isr_1124707_43.stderr parsl.dataflow.dflow INFO: Task 291 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7165e97dd3ad7d5908e961316175ddf1 with result from task 291 parsl.dataflow.dflow DEBUG: Task 291 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 292 has memoization hash fa7cbf0f85b895f612f9413827df5257 parsl.dataflow.memoization INFO: Task 292 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cc360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 292 try 0 launched on executor multi with executor id 420 parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/f00b299f-8934-4119-b407-a12e9183a60a_characterizeImage_1124707_39.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/f00b299f-8934-4119-b407-a12e9183a60a_characterizeImage_1124707_39.stderr parsl.dataflow.dflow INFO: Standard output for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/aff9fb28-b99f-4f2f-a1a1-b722e6b498cc_isr_1124707_39.stdout parsl.dataflow.dflow INFO: Standard error for task 291 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/aff9fb28-b99f-4f2f-a1a1-b722e6b498cc_isr_1124707_39.stderr parsl.dataflow.dflow INFO: Task 309 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7b0fc5cd4574fba5db424776a162160d with result from task 309 parsl.dataflow.dflow DEBUG: Task 309 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 310 has memoization hash 2b20d2d4199e5bb6b2b8678cb8b6bc8f parsl.dataflow.memoization INFO: Task 310 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cd620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 310 try 0 launched on executor multi with executor id 421 parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/0b027ecc-8728-4bd2-9ae0-b99553738615_characterizeImage_1124704_39.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/0b027ecc-8728-4bd2-9ae0-b99553738615_characterizeImage_1124704_39.stderr parsl.dataflow.dflow INFO: Standard output for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6a057159-5fc8-4bbf-984b-581889166c82_isr_1124704_39.stdout parsl.dataflow.dflow INFO: Standard error for task 309 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6a057159-5fc8-4bbf-984b-581889166c82_isr_1124704_39.stderr parsl.dataflow.dflow INFO: Task 297 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78df74af96ddf3bd44c71647646f834c with result from task 297 parsl.dataflow.dflow DEBUG: Task 297 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 298 has memoization hash 98162062fc27142b01ad1e2c3efcd361 parsl.dataflow.memoization INFO: Task 298 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cc9a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 298 try 0 launched on executor multi with executor id 422 parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0e4487b8-7d83-4560-9fa9-62cf50006c09_characterizeImage_1124828_43.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0e4487b8-7d83-4560-9fa9-62cf50006c09_characterizeImage_1124828_43.stderr parsl.dataflow.dflow INFO: Standard output for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/eb36a66d-ef2d-480c-99e5-327ef2e171d9_isr_1124828_43.stdout parsl.dataflow.dflow INFO: Standard error for task 297 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/eb36a66d-ef2d-480c-99e5-327ef2e171d9_isr_1124828_43.stderr parsl.dataflow.dflow INFO: Task 303 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f1c533ac9b38b9f79813699e77f5732 with result from task 303 parsl.dataflow.dflow DEBUG: Task 303 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 304 has memoization hash 4c0a5cc455347cd2e2a93a2e4efeb549 parsl.dataflow.memoization INFO: Task 304 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ccfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 304 try 0 launched on executor multi with executor id 423 parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/a614cf75-b770-4492-9dd4-19da8401e4a4_characterizeImage_1124831_14.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/a614cf75-b770-4492-9dd4-19da8401e4a4_characterizeImage_1124831_14.stderr parsl.dataflow.dflow INFO: Standard output for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/63fb24f9-bf4a-403f-9e77-348988cb3880_isr_1124831_14.stdout parsl.dataflow.dflow INFO: Standard error for task 303 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/63fb24f9-bf4a-403f-9e77-348988cb3880_isr_1124831_14.stderr parsl.dataflow.dflow INFO: Task 312 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ec6a0b9cd91ac3ebc227c76565cc66c with result from task 312 parsl.dataflow.dflow DEBUG: Task 312 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 313 has memoization hash 7da54d1fb39df461903f51d053b3d85b parsl.dataflow.memoization INFO: Task 313 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cd940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 313 try 0 launched on executor multi with executor id 424 parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f3ccc0fd-66ac-44dc-8b24-982d8c05ea72_characterizeImage_1124831_1.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f3ccc0fd-66ac-44dc-8b24-982d8c05ea72_characterizeImage_1124831_1.stderr parsl.dataflow.dflow INFO: Standard output for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/3cc4896d-3102-4af8-88eb-34a474177ac6_isr_1124831_1.stdout parsl.dataflow.dflow INFO: Standard error for task 312 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/3cc4896d-3102-4af8-88eb-34a474177ac6_isr_1124831_1.stderr parsl.dataflow.dflow INFO: Task 294 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5169ea28f5074dabec5ac5e03bb77f6a with result from task 294 parsl.dataflow.dflow DEBUG: Task 294 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 295 has memoization hash fcb96002a96a69971f9a931e17bd9a88 parsl.dataflow.memoization INFO: Task 295 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cc680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 295 try 0 launched on executor multi with executor id 425 parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/b9d9326f-e440-4a8c-b5a3-11e87fc74a38_characterizeImage_1124707_46.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/b9d9326f-e440-4a8c-b5a3-11e87fc74a38_characterizeImage_1124707_46.stderr parsl.dataflow.dflow INFO: Standard output for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/93e46d19-5a37-4bf4-a379-94db4be1b2ad_isr_1124707_46.stdout parsl.dataflow.dflow INFO: Standard error for task 294 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/93e46d19-5a37-4bf4-a379-94db4be1b2ad_isr_1124707_46.stderr parsl.dataflow.dflow INFO: Task 399 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f774b27eb14e91e0845e39896b837e13 with result from task 399 parsl.dataflow.dflow DEBUG: Task 399 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 400 has memoization hash eeabde7777b8da66a82d16b9c1dd3c9d parsl.dataflow.memoization INFO: Task 400 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 400 try 0 launched on executor multi with executor id 426 parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d438c735-ed49-47fa-a595-a7dd50f442e8_characterizeImage_1124704_46.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d438c735-ed49-47fa-a595-a7dd50f442e8_characterizeImage_1124704_46.stderr parsl.dataflow.dflow INFO: Standard output for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ba9d6043-a928-4dc2-a4cd-b0148deefd24_isr_1124704_46.stdout parsl.dataflow.dflow INFO: Standard error for task 399 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ba9d6043-a928-4dc2-a4cd-b0148deefd24_isr_1124704_46.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:52 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 375 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d7c5f9e0e2257f2b3d61ccb300208782 with result from task 375 parsl.dataflow.dflow DEBUG: Task 375 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 376 has memoization hash e286d664317dab90959e5fe3ac8d80b1 parsl.dataflow.memoization INFO: Task 376 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 376 try 0 launched on executor multi with executor id 427 parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b51f50b9-0407-4622-bec2-874c0813e35e_characterizeImage_1124704_26.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b51f50b9-0407-4622-bec2-874c0813e35e_characterizeImage_1124704_26.stderr parsl.dataflow.dflow INFO: Standard output for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/b8d4d2a9-d54a-4dfa-ae41-02010fed3343_isr_1124704_26.stdout parsl.dataflow.dflow INFO: Standard error for task 375 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/b8d4d2a9-d54a-4dfa-ae41-02010fed3343_isr_1124704_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:53 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 369 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4fab9ded05f97de7469eed738e2bbf28 with result from task 369 parsl.dataflow.dflow DEBUG: Task 369 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash f2e3ec23c7297a32110a9ffdedcd1967 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 0 launched on executor multi with executor id 428 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/45b82ec0-bfba-47fd-baa4-5f7cbe3dceab_characterizeImage_1124831_61.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/45b82ec0-bfba-47fd-baa4-5f7cbe3dceab_characterizeImage_1124831_61.stderr parsl.dataflow.dflow INFO: Standard output for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/469ffe4a-a533-4333-bac3-e6ba2009c98b_isr_1124831_61.stdout parsl.dataflow.dflow INFO: Standard error for task 369 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/469ffe4a-a533-4333-bac3-e6ba2009c98b_isr_1124831_61.stderr parsl.dataflow.dflow INFO: Task 387 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ca0662438a2ef996ebf8e76258a7fbe with result from task 387 parsl.dataflow.dflow DEBUG: Task 387 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 388 has memoization hash 52827528aa2758d1b71f76393e8e41b6 parsl.dataflow.memoization INFO: Task 388 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ea0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 388 try 0 launched on executor multi with executor id 429 parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/641883f6-2955-4b61-b638-a6074541fa8f_characterizeImage_1124704_7.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/641883f6-2955-4b61-b638-a6074541fa8f_characterizeImage_1124704_7.stderr parsl.dataflow.dflow INFO: Standard output for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6566e522-7234-4c9e-b329-fdb4ef392647_isr_1124704_7.stdout parsl.dataflow.dflow INFO: Standard error for task 387 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6566e522-7234-4c9e-b329-fdb4ef392647_isr_1124704_7.stderr parsl.dataflow.dflow INFO: Task 306 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c2c678c8bf9ba7ccfb29403215ba7564 with result from task 306 parsl.dataflow.dflow DEBUG: Task 306 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 307 has memoization hash 3d1f8f8f53032297b9fb9093e8646b4e parsl.dataflow.memoization INFO: Task 307 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cd300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 307 try 0 launched on executor multi with executor id 430 parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e8dd0d7f-f643-4245-8d9c-32cc3e1c9ce6_characterizeImage_1124707_60.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e8dd0d7f-f643-4245-8d9c-32cc3e1c9ce6_characterizeImage_1124707_60.stderr parsl.dataflow.dflow INFO: Standard output for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/2295dd8a-1684-4c1b-b973-d468334901b0_isr_1124707_60.stdout parsl.dataflow.dflow INFO: Standard error for task 306 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/2295dd8a-1684-4c1b-b973-d468334901b0_isr_1124707_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:54 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 423 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc70faa7b92b1e233037ad1a5f035202 with result from task 423 parsl.dataflow.dflow DEBUG: Task 423 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 424 has memoization hash ae974ea880856a1670603e1509ecad8b parsl.dataflow.memoization INFO: Task 424 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 424 try 0 launched on executor multi with executor id 431 parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bd31ce0e-38b1-4eac-a126-8ee5c782a896_characterizeImage_1124704_41.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bd31ce0e-38b1-4eac-a126-8ee5c782a896_characterizeImage_1124704_41.stderr parsl.dataflow.dflow INFO: Standard output for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e172a54d-5269-415b-9717-68fb36974086_isr_1124704_41.stdout parsl.dataflow.dflow INFO: Standard error for task 423 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e172a54d-5269-415b-9717-68fb36974086_isr_1124704_41.stderr parsl.dataflow.dflow INFO: Task 402 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63857381bcb84ade73970f53a78d0a2c with result from task 402 parsl.dataflow.dflow DEBUG: Task 402 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 403 has memoization hash c861a745cbca9e9a5b8317c5535fd442 parsl.dataflow.memoization INFO: Task 403 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 403 try 0 launched on executor multi with executor id 432 parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/6eea15c4-6086-48b5-916b-f75b523bc14d_characterizeImage_1124707_15.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/6eea15c4-6086-48b5-916b-f75b523bc14d_characterizeImage_1124707_15.stderr parsl.dataflow.dflow INFO: Standard output for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/a24ef897-9c7b-4b24-ad33-085ac37d5a1b_isr_1124707_15.stdout parsl.dataflow.dflow INFO: Standard error for task 402 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/a24ef897-9c7b-4b24-ad33-085ac37d5a1b_isr_1124707_15.stderr parsl.dataflow.dflow INFO: Task 417 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71191b644a7f5c5025b6ae90159ed504 with result from task 417 parsl.dataflow.dflow DEBUG: Task 417 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 418 has memoization hash 6ce646a1440c49d4ae2d1029af931870 parsl.dataflow.memoization INFO: Task 418 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981d6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 418 try 0 launched on executor multi with executor id 433 parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/09be60e8-3e8b-454c-805f-c64a47f62445_characterizeImage_1124710_53.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/09be60e8-3e8b-454c-805f-c64a47f62445_characterizeImage_1124710_53.stderr parsl.dataflow.dflow INFO: Standard output for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/d1f2a26c-4b56-4a8a-958a-43f5553a7700_isr_1124710_53.stdout parsl.dataflow.dflow INFO: Standard error for task 417 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/d1f2a26c-4b56-4a8a-958a-43f5553a7700_isr_1124710_53.stderr parsl.dataflow.dflow INFO: Task 420 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ddca065e0496b1c902f7c3b8f7c76754 with result from task 420 parsl.dataflow.dflow DEBUG: Task 420 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 421 has memoization hash 10444017c6fcdfee44e03cfe8e0dd470 parsl.dataflow.memoization INFO: Task 421 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 421 try 0 launched on executor multi with executor id 434 parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/49857fee-66ca-41f3-9d0b-ab61610a1738_characterizeImage_1124828_16.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/49857fee-66ca-41f3-9d0b-ab61610a1738_characterizeImage_1124828_16.stderr parsl.dataflow.dflow INFO: Standard output for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/3dd8b0e8-e8db-4311-907e-63039da12afd_isr_1124828_16.stdout parsl.dataflow.dflow INFO: Standard error for task 420 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/3dd8b0e8-e8db-4311-907e-63039da12afd_isr_1124828_16.stderr parsl.dataflow.dflow INFO: Task 396 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4bcf22379b32637d1f9ebf199d3c2d5f with result from task 396 parsl.dataflow.dflow DEBUG: Task 396 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 397 has memoization hash dc0d34c19bbbd7f34c6d01299dd8bab8 parsl.dataflow.memoization INFO: Task 397 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ea840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 397 try 0 launched on executor multi with executor id 435 parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/15bb7c4f-475f-4548-a0d0-4bcc0a2e34b3_characterizeImage_1124828_46.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/15bb7c4f-475f-4548-a0d0-4bcc0a2e34b3_characterizeImage_1124828_46.stderr parsl.dataflow.dflow INFO: Standard output for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/d15c45b2-66be-46d5-b098-a324f0293c85_isr_1124828_46.stdout parsl.dataflow.dflow INFO: Standard error for task 396 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/d15c45b2-66be-46d5-b098-a324f0293c85_isr_1124828_46.stderr parsl.dataflow.dflow INFO: Task 378 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e297212bddc52d34bb605411ed853c8 with result from task 378 parsl.dataflow.dflow DEBUG: Task 378 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 379 has memoization hash a27504cde7359b916a3aa0ad9be5ac11 parsl.dataflow.memoization INFO: Task 379 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ebba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 379 try 0 launched on executor multi with executor id 436 parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/59aba1fe-a453-4fcc-90f5-87c5611a72e3_characterizeImage_1124704_45.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/59aba1fe-a453-4fcc-90f5-87c5611a72e3_characterizeImage_1124704_45.stderr parsl.dataflow.dflow INFO: Standard output for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/1464ff7b-dde9-4e3c-ab63-905e3326595e_isr_1124704_45.stdout parsl.dataflow.dflow INFO: Standard error for task 378 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/1464ff7b-dde9-4e3c-ab63-905e3326595e_isr_1124704_45.stderr parsl.dataflow.dflow INFO: Task 384 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b908be95ee05f6891e8a961e640f402 with result from task 384 parsl.dataflow.dflow DEBUG: Task 384 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 385 has memoization hash a2bb8a9d39c7bf654190ca2c499f46fc parsl.dataflow.memoization INFO: Task 385 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ea200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 385 try 0 launched on executor multi with executor id 437 parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/95303cea-2320-4c24-b7c0-9c5c3235eaaf_characterizeImage_1124828_48.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/95303cea-2320-4c24-b7c0-9c5c3235eaaf_characterizeImage_1124828_48.stderr parsl.dataflow.dflow INFO: Standard output for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/a900e9de-7224-449a-a5a0-482e817dce4f_isr_1124828_48.stdout parsl.dataflow.dflow INFO: Standard error for task 384 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/a900e9de-7224-449a-a5a0-482e817dce4f_isr_1124828_48.stderr parsl.dataflow.dflow INFO: Task 381 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e0f01160cde06eb3dfd2569e1d39600 with result from task 381 parsl.dataflow.dflow DEBUG: Task 381 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 382 has memoization hash c5b688a838dd4142419c4a91b242bfe1 parsl.dataflow.memoization INFO: Task 382 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 382 try 0 launched on executor multi with executor id 438 parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/69ff441e-0144-40a3-b07c-2b686bfd27f1_characterizeImage_1124710_51.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/69ff441e-0144-40a3-b07c-2b686bfd27f1_characterizeImage_1124710_51.stderr parsl.dataflow.dflow INFO: Standard output for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/7d10d2ae-4399-4db2-b4a9-fefb0f96d598_isr_1124710_51.stdout parsl.dataflow.dflow INFO: Standard error for task 381 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/7d10d2ae-4399-4db2-b4a9-fefb0f96d598_isr_1124710_51.stderr parsl.dataflow.dflow INFO: Task 429 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf449ea4e94479c89074f7c3404b09a8 with result from task 429 parsl.dataflow.dflow DEBUG: Task 429 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 430 has memoization hash 071dc5911b0a5e18935e448ffa6d35a9 parsl.dataflow.memoization INFO: Task 430 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981cfe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 430 try 0 launched on executor multi with executor id 439 parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/7b9795e0-d915-4ed7-aef8-aab9fae36e94_characterizeImage_1124707_42.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/7b9795e0-d915-4ed7-aef8-aab9fae36e94_characterizeImage_1124707_42.stderr parsl.dataflow.dflow INFO: Standard output for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/9b9db11d-5414-4e5c-8cac-12f3ebafb89b_isr_1124707_42.stdout parsl.dataflow.dflow INFO: Standard error for task 429 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/9b9db11d-5414-4e5c-8cac-12f3ebafb89b_isr_1124707_42.stderr parsl.dataflow.dflow INFO: Task 444 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e73560942072fdc530ad0f49396cd5c4 with result from task 444 parsl.dataflow.dflow DEBUG: Task 444 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 445 has memoization hash a3e7ff70d6f268359b387d70484e2a17 parsl.dataflow.memoization INFO: Task 445 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 445 try 0 launched on executor multi with executor id 440 parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0c471b40-2636-4ba2-8b0e-ead64edef852_characterizeImage_1124828_45.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0c471b40-2636-4ba2-8b0e-ead64edef852_characterizeImage_1124828_45.stderr parsl.dataflow.dflow INFO: Standard output for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/76c2964e-4bd2-4613-9c6f-48b8b78c8af8_isr_1124828_45.stdout parsl.dataflow.dflow INFO: Standard error for task 444 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/76c2964e-4bd2-4613-9c6f-48b8b78c8af8_isr_1124828_45.stderr parsl.dataflow.dflow INFO: Task 435 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 654131854e800e1c7fa0799f5ee8d702 with result from task 435 parsl.dataflow.dflow DEBUG: Task 435 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 436 has memoization hash be37226d2cd7cb27e0defc51d4280e40 parsl.dataflow.memoization INFO: Task 436 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 436 try 0 launched on executor multi with executor id 441 parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5c7859d8-7cf6-4dc8-90e7-8dc332d53fa2_characterizeImage_1124831_44.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5c7859d8-7cf6-4dc8-90e7-8dc332d53fa2_characterizeImage_1124831_44.stderr parsl.dataflow.dflow INFO: Standard output for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/0ffaa047-75c0-4815-809c-ee1a2d30e2fa_isr_1124831_44.stdout parsl.dataflow.dflow INFO: Standard error for task 435 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/0ffaa047-75c0-4815-809c-ee1a2d30e2fa_isr_1124831_44.stderr parsl.dataflow.dflow INFO: Task 390 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6bc097a40f2c693e575d93debb33f3d with result from task 390 parsl.dataflow.dflow DEBUG: Task 390 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 391 has memoization hash f43ab3802ee4349672de393db6beaab0 parsl.dataflow.memoization INFO: Task 391 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981d940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 391 try 0 launched on executor multi with executor id 442 parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/c56d0a4d-d0ef-4586-8db1-d774ce530d85_characterizeImage_1124828_30.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/c56d0a4d-d0ef-4586-8db1-d774ce530d85_characterizeImage_1124828_30.stderr parsl.dataflow.dflow INFO: Standard output for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/e3eef410-6b4a-439c-86d3-69d761f24e45_isr_1124828_30.stdout parsl.dataflow.dflow INFO: Standard error for task 390 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/e3eef410-6b4a-439c-86d3-69d761f24e45_isr_1124828_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:56 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 432 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81f08a7ed3d8259fdb5fae7492e5aa4a with result from task 432 parsl.dataflow.dflow DEBUG: Task 432 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 433 has memoization hash 807aafe0f8994d790c578ef441f19432 parsl.dataflow.memoization INFO: Task 433 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981ccc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 433 try 0 launched on executor multi with executor id 443 parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/d5610802-5adf-429b-bf0a-f6f6632a1f16_characterizeImage_1124828_36.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/d5610802-5adf-429b-bf0a-f6f6632a1f16_characterizeImage_1124828_36.stderr parsl.dataflow.dflow INFO: Standard output for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8df0d3b8-6710-443d-8726-95db9409d197_isr_1124828_36.stdout parsl.dataflow.dflow INFO: Standard error for task 432 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8df0d3b8-6710-443d-8726-95db9409d197_isr_1124828_36.stderr parsl.dataflow.dflow INFO: Task 414 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a694b0ea574cae329ac9e3d1eac5cf88 with result from task 414 parsl.dataflow.dflow DEBUG: Task 414 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 415 has memoization hash 1c91191510c1daec301e5bd6be96f1dd parsl.dataflow.memoization INFO: Task 415 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ebce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 415 try 0 launched on executor multi with executor id 444 parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/c1ec030a-0e8d-4a22-bbc7-444d6cea8319_characterizeImage_1124707_9.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/c1ec030a-0e8d-4a22-bbc7-444d6cea8319_characterizeImage_1124707_9.stderr parsl.dataflow.dflow INFO: Standard output for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/69dc5d70-3b7f-4197-9e5f-f812aa85a6ac_isr_1124707_9.stdout parsl.dataflow.dflow INFO: Standard error for task 414 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/69dc5d70-3b7f-4197-9e5f-f812aa85a6ac_isr_1124707_9.stderr parsl.dataflow.dflow INFO: Task 441 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1de8a5a4b291c451e4f9438a2b50a59 with result from task 441 parsl.dataflow.dflow DEBUG: Task 441 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 442 has memoization hash 73615d31b8ad90cfad990143713d51fd parsl.dataflow.memoization INFO: Task 442 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981e3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 442 try 0 launched on executor multi with executor id 445 parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d44a6066-7a9b-4d81-95ac-65f9359a67bd_characterizeImage_1124704_10.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d44a6066-7a9b-4d81-95ac-65f9359a67bd_characterizeImage_1124704_10.stderr parsl.dataflow.dflow INFO: Standard output for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ce5d6539-ec19-4f97-86f0-f55331c83ec1_isr_1124704_10.stdout parsl.dataflow.dflow INFO: Standard error for task 441 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/ce5d6539-ec19-4f97-86f0-f55331c83ec1_isr_1124704_10.stderr parsl.dataflow.dflow INFO: Task 411 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cac550fe12448aca753b09dec8ebbeb9 with result from task 411 parsl.dataflow.dflow DEBUG: Task 411 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 412 has memoization hash 87e350ccbc95679372874289b05ca2bb parsl.dataflow.memoization INFO: Task 412 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 412 try 0 launched on executor multi with executor id 446 parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e3229bb8-b47c-4bf6-b459-76ae366746e0_characterizeImage_1124707_5.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e3229bb8-b47c-4bf6-b459-76ae366746e0_characterizeImage_1124707_5.stderr parsl.dataflow.dflow INFO: Standard output for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/397495d7-aa4b-49a7-a084-4a117af34e36_isr_1124707_5.stdout parsl.dataflow.dflow INFO: Standard error for task 411 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/397495d7-aa4b-49a7-a084-4a117af34e36_isr_1124707_5.stderr parsl.dataflow.dflow INFO: Task 393 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 183b038a804b518aa746eb0381b5d923 with result from task 393 parsl.dataflow.dflow DEBUG: Task 393 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 394 has memoization hash 29f1f247b39ded91c94eb3e58ecf1b54 parsl.dataflow.memoization INFO: Task 394 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 394 try 0 launched on executor multi with executor id 447 parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7fa8bf0e-1d8a-4caf-8b82-3cc9377a91f7_characterizeImage_1124710_48.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7fa8bf0e-1d8a-4caf-8b82-3cc9377a91f7_characterizeImage_1124710_48.stderr parsl.dataflow.dflow INFO: Standard output for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/efee96ae-33ba-447a-8a39-0be85d628f52_isr_1124710_48.stdout parsl.dataflow.dflow INFO: Standard error for task 393 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/efee96ae-33ba-447a-8a39-0be85d628f52_isr_1124710_48.stderr parsl.dataflow.dflow INFO: Task 408 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f84bf233c652f198448419ea90cddfb9 with result from task 408 parsl.dataflow.dflow DEBUG: Task 408 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 409 has memoization hash e00424e7e1c0f46fc40e86f265b5d56d parsl.dataflow.memoization INFO: Task 409 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 409 try 0 launched on executor multi with executor id 448 parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/fdb91c8c-087c-44ad-94b3-4603e22ed44a_characterizeImage_1124707_48.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/fdb91c8c-087c-44ad-94b3-4603e22ed44a_characterizeImage_1124707_48.stderr parsl.dataflow.dflow INFO: Standard output for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/de53283b-1af0-4ab8-b87b-8c1b68f01d7f_isr_1124707_48.stdout parsl.dataflow.dflow INFO: Standard error for task 408 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/de53283b-1af0-4ab8-b87b-8c1b68f01d7f_isr_1124707_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:58 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 438 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac14081fff13aa487c909a6a8748fea2 with result from task 438 parsl.dataflow.dflow DEBUG: Task 438 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 439 has memoization hash d360466ad3ec7520e08b1b98cfa7a967 parsl.dataflow.memoization INFO: Task 439 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 439 try 0 launched on executor multi with executor id 449 parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/b0d26e0e-ef4c-4625-aedf-57b26b21562c_characterizeImage_1124828_23.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/b0d26e0e-ef4c-4625-aedf-57b26b21562c_characterizeImage_1124828_23.stderr parsl.dataflow.dflow INFO: Standard output for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/4119fea4-5196-4ae2-b2f1-a71a9bcfcc86_isr_1124828_23.stdout parsl.dataflow.dflow INFO: Standard error for task 438 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/4119fea4-5196-4ae2-b2f1-a71a9bcfcc86_isr_1124828_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:26:59 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 405 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f4f38c250ec1bb3f71089d241ae8d98 with result from task 405 parsl.dataflow.dflow DEBUG: Task 405 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 406 has memoization hash fc53327fe930ac7f26660c2f3c1e7cb2 parsl.dataflow.memoization INFO: Task 406 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eb380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 406 try 0 launched on executor multi with executor id 450 parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0b0d8ad1-3a1e-4966-86ca-de170b060f01_characterizeImage_1124828_37.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0b0d8ad1-3a1e-4966-86ca-de170b060f01_characterizeImage_1124828_37.stderr parsl.dataflow.dflow INFO: Standard output for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8338e9cd-d266-4aae-a3b0-19dd92053dd6_isr_1124828_37.stdout parsl.dataflow.dflow INFO: Standard error for task 405 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8338e9cd-d266-4aae-a3b0-19dd92053dd6_isr_1124828_37.stderr parsl.dataflow.dflow INFO: Task 426 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24f1088288399d51f087fd33e4e138a0 with result from task 426 parsl.dataflow.dflow DEBUG: Task 426 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 427 has memoization hash 999a8be926a22baba70afcd019c688d8 parsl.dataflow.memoization INFO: Task 427 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 427 try 0 launched on executor multi with executor id 451 parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/d621ef3f-f21a-4526-906c-927a96a1de6a_characterizeImage_1124831_60.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/d621ef3f-f21a-4526-906c-927a96a1de6a_characterizeImage_1124831_60.stderr parsl.dataflow.dflow INFO: Standard output for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/e4ab1bcb-25f3-46a0-b1a7-e9540f54665d_isr_1124831_60.stdout parsl.dataflow.dflow INFO: Standard error for task 426 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/e4ab1bcb-25f3-46a0-b1a7-e9540f54665d_isr_1124831_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:00 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:01 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 492 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1116e3df0a5dbbbd48bbd0dec4db0a73 with result from task 492 parsl.dataflow.dflow DEBUG: Task 492 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 493 has memoization hash cc79aa1cea4d9fef7617c42a8bde89e8 parsl.dataflow.memoization INFO: Task 493 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529851120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 493 try 0 launched on executor multi with executor id 452 parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0caafbae-1fa3-4c1e-8169-750ed23ea692_characterizeImage_1124828_20.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0caafbae-1fa3-4c1e-8169-750ed23ea692_characterizeImage_1124828_20.stderr parsl.dataflow.dflow INFO: Standard output for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/44353e88-1e3e-498c-a41b-fbff07cd7786_isr_1124828_20.stdout parsl.dataflow.dflow INFO: Standard error for task 492 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/44353e88-1e3e-498c-a41b-fbff07cd7786_isr_1124828_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:02 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 450 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f59c260d3c0225241598825ead89d2f4 with result from task 450 parsl.dataflow.dflow DEBUG: Task 450 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 451 has memoization hash 7da73a4e10559259f6a0dc1f92ff3d5f parsl.dataflow.memoization INFO: Task 451 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981eb60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 451 try 0 launched on executor multi with executor id 453 parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/80096367-fa6c-4c17-90ac-046d6403c4c2_characterizeImage_1124710_60.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/80096367-fa6c-4c17-90ac-046d6403c4c2_characterizeImage_1124710_60.stderr parsl.dataflow.dflow INFO: Standard output for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/af034429-28fc-4a94-9320-ea1c3d37d6e7_isr_1124710_60.stdout parsl.dataflow.dflow INFO: Standard error for task 450 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/af034429-28fc-4a94-9320-ea1c3d37d6e7_isr_1124710_60.stderr parsl.dataflow.dflow INFO: Task 498 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bec64eedc925f384608ed93cf1c2985 with result from task 498 parsl.dataflow.dflow DEBUG: Task 498 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 499 has memoization hash fb1f86e07a582ad63fc3a1f2081cd70a parsl.dataflow.memoization INFO: Task 499 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298516c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 499 try 0 launched on executor multi with executor id 454 parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f8f9f41a-95e2-4e7e-a477-3084d55627fa_characterizeImage_1124831_13.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f8f9f41a-95e2-4e7e-a477-3084d55627fa_characterizeImage_1124831_13.stderr parsl.dataflow.dflow INFO: Standard output for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/6146528f-560e-418d-8d1e-31b90e334f9c_isr_1124831_13.stdout parsl.dataflow.dflow INFO: Standard error for task 498 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/6146528f-560e-418d-8d1e-31b90e334f9c_isr_1124831_13.stderr parsl.dataflow.dflow INFO: Task 459 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0dc03bae600f1cbaedec6fc48cc97e3 with result from task 459 parsl.dataflow.dflow DEBUG: Task 459 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 460 has memoization hash f24ac40c8e757bdfebb81bdca0130d5f parsl.dataflow.memoization INFO: Task 460 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 460 try 0 launched on executor multi with executor id 455 parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/188137e2-e3bc-4d8e-a172-e2d2e6212b31_characterizeImage_1124710_52.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/188137e2-e3bc-4d8e-a172-e2d2e6212b31_characterizeImage_1124710_52.stderr parsl.dataflow.dflow INFO: Standard output for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/6c37fa58-4db2-4ff2-a65b-36c5a747c41a_isr_1124710_52.stdout parsl.dataflow.dflow INFO: Standard error for task 459 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/6c37fa58-4db2-4ff2-a65b-36c5a747c41a_isr_1124710_52.stderr parsl.dataflow.dflow INFO: Task 501 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c91f404907ff9cad257444aa291eda34 with result from task 501 parsl.dataflow.dflow DEBUG: Task 501 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 502 has memoization hash f27bee7ef348d7e869f146dea0a0b79a parsl.dataflow.memoization INFO: Task 502 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298513a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 502 try 0 launched on executor multi with executor id 456 parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8574765d-1771-4f9c-a07d-3b066223e9bc_characterizeImage_1124707_53.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8574765d-1771-4f9c-a07d-3b066223e9bc_characterizeImage_1124707_53.stderr parsl.dataflow.dflow INFO: Standard output for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/2887118a-7013-432c-a361-6fbd9c613ac3_isr_1124707_53.stdout parsl.dataflow.dflow INFO: Standard error for task 501 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/2887118a-7013-432c-a361-6fbd9c613ac3_isr_1124707_53.stderr parsl.dataflow.dflow INFO: Task 519 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ab5d7fd1dd66e81541d4b0b775d5bcd with result from task 519 parsl.dataflow.dflow DEBUG: Task 519 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 520 has memoization hash dc193fb503074e0b0b33f8b4336db520 parsl.dataflow.memoization INFO: Task 520 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529853380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 520 try 0 launched on executor multi with executor id 457 parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/90d12b2c-62d1-4afa-a459-e52083eafd74_characterizeImage_1124831_49.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/90d12b2c-62d1-4afa-a459-e52083eafd74_characterizeImage_1124831_49.stderr parsl.dataflow.dflow INFO: Standard output for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/1dc6dea3-9d6c-4993-bc75-a8700590f7ac_isr_1124831_49.stdout parsl.dataflow.dflow INFO: Standard error for task 519 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/1dc6dea3-9d6c-4993-bc75-a8700590f7ac_isr_1124831_49.stderr parsl.dataflow.dflow INFO: Task 486 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b1b9622d75965056d21ac86108cca312 with result from task 486 parsl.dataflow.dflow DEBUG: Task 486 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 487 has memoization hash a7a495e79409ed28547de4ae69537376 parsl.dataflow.memoization INFO: Task 487 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529850ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 487 try 0 launched on executor multi with executor id 458 parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/08ad96ab-3f9b-4eb0-b2df-ed72374bd680_characterizeImage_1124707_51.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/08ad96ab-3f9b-4eb0-b2df-ed72374bd680_characterizeImage_1124707_51.stderr parsl.dataflow.dflow INFO: Standard output for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/adf876a1-27de-4f19-93c8-20b6b29e8301_isr_1124707_51.stdout parsl.dataflow.dflow INFO: Standard error for task 486 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/adf876a1-27de-4f19-93c8-20b6b29e8301_isr_1124707_51.stderr parsl.dataflow.dflow INFO: Task 447 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f7e237e0d22e5b9d0c65dfe3ae81c21 with result from task 447 parsl.dataflow.dflow DEBUG: Task 447 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 448 has memoization hash c88745a4ce2964340d0c6a78958aea7c parsl.dataflow.memoization INFO: Task 448 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ea700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 448 try 0 launched on executor multi with executor id 459 parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3a3e8de2-f12e-4865-b52a-5156b32f3032_characterizeImage_1124707_54.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3a3e8de2-f12e-4865-b52a-5156b32f3032_characterizeImage_1124707_54.stderr parsl.dataflow.dflow INFO: Standard output for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/3c33c95c-a77e-4448-b50c-696b79df0161_isr_1124707_54.stdout parsl.dataflow.dflow INFO: Standard error for task 447 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/3c33c95c-a77e-4448-b50c-696b79df0161_isr_1124707_54.stderr parsl.dataflow.dflow INFO: Task 489 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b072cae0cc916837f19a575f0f3362de with result from task 489 parsl.dataflow.dflow DEBUG: Task 489 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 490 has memoization hash b7aaaa66fc9eec519d39aeed3825ad75 parsl.dataflow.memoization INFO: Task 490 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529850e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 490 try 0 launched on executor multi with executor id 460 parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f7d5e89a-6d50-479b-bfa7-697ceafdf224_characterizeImage_1124831_32.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f7d5e89a-6d50-479b-bfa7-697ceafdf224_characterizeImage_1124831_32.stderr parsl.dataflow.dflow INFO: Standard output for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/09d560f6-7767-42c8-8d98-887fc49c25b5_isr_1124831_32.stdout parsl.dataflow.dflow INFO: Standard error for task 489 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/09d560f6-7767-42c8-8d98-887fc49c25b5_isr_1124831_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:03 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 456 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86ede9c59684c015d512af04b5d50ab3 with result from task 456 parsl.dataflow.dflow DEBUG: Task 456 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 457 has memoization hash ea4a67b34158a9da41c02b323b6319f9 parsl.dataflow.memoization INFO: Task 457 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 457 try 0 launched on executor multi with executor id 461 parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/e0d0c77f-0784-4ed2-98c8-c223582bfbf4_characterizeImage_1124828_35.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/e0d0c77f-0784-4ed2-98c8-c223582bfbf4_characterizeImage_1124828_35.stderr parsl.dataflow.dflow INFO: Standard output for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/2d42e3b2-a88b-4fed-bdcd-c4fb0ebf644c_isr_1124828_35.stdout parsl.dataflow.dflow INFO: Standard error for task 456 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/2d42e3b2-a88b-4fed-bdcd-c4fb0ebf644c_isr_1124828_35.stderr parsl.dataflow.dflow INFO: Task 504 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88ffaabb9024c1f25e86a0a2bc279f77 with result from task 504 parsl.dataflow.dflow DEBUG: Task 504 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 505 has memoization hash 2fc1b33a37a956cd28eaf34fb9c728b0 parsl.dataflow.memoization INFO: Task 505 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298525c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 505 try 0 launched on executor multi with executor id 462 parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/1b45cdee-18d5-4284-82ae-de8e8a6a1b16_characterizeImage_1124831_4.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/1b45cdee-18d5-4284-82ae-de8e8a6a1b16_characterizeImage_1124831_4.stderr parsl.dataflow.dflow INFO: Standard output for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/25d89f12-1d48-4bd8-8e8b-6651fbb7cd67_isr_1124831_4.stdout parsl.dataflow.dflow INFO: Standard error for task 504 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/25d89f12-1d48-4bd8-8e8b-6651fbb7cd67_isr_1124831_4.stderr parsl.dataflow.dflow INFO: Task 483 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6de05e5758c75916ff21f5be78600789 with result from task 483 parsl.dataflow.dflow DEBUG: Task 483 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 484 has memoization hash 8d062f724d22b1973c557d55c9bb038a parsl.dataflow.memoization INFO: Task 484 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298507c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 484 try 0 launched on executor multi with executor id 463 parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7006ff7b-5f8a-4b99-b95a-9d4c9ed70c83_characterizeImage_1124710_34.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7006ff7b-5f8a-4b99-b95a-9d4c9ed70c83_characterizeImage_1124710_34.stderr parsl.dataflow.dflow INFO: Standard output for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/993d57d2-ff78-4686-9911-401ce7cca79e_isr_1124710_34.stdout parsl.dataflow.dflow INFO: Standard error for task 483 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/993d57d2-ff78-4686-9911-401ce7cca79e_isr_1124710_34.stderr parsl.dataflow.dflow INFO: Task 480 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry efe23a340771169f9ab2a7bd4f6c0993 with result from task 480 parsl.dataflow.dflow DEBUG: Task 480 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 481 has memoization hash 7e1c374dc5eada5bed4c13be03706bfe parsl.dataflow.memoization INFO: Task 481 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298504a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 481 try 0 launched on executor multi with executor id 464 parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/354f14e4-6ccf-46e7-b39c-14a4a741f111_characterizeImage_1124831_16.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/354f14e4-6ccf-46e7-b39c-14a4a741f111_characterizeImage_1124831_16.stderr parsl.dataflow.dflow INFO: Standard output for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/ad8af73f-e0ac-441f-bf89-d6ca9733cacd_isr_1124831_16.stdout parsl.dataflow.dflow INFO: Standard error for task 480 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/ad8af73f-e0ac-441f-bf89-d6ca9733cacd_isr_1124831_16.stderr parsl.dataflow.dflow INFO: Task 510 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7798f95312077a220093f9940937889 with result from task 510 parsl.dataflow.dflow DEBUG: Task 510 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 511 has memoization hash 8c2b337244c0ca1bec8515f1e56f89c9 parsl.dataflow.memoization INFO: Task 511 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852a20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 511 try 0 launched on executor multi with executor id 465 parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/c217ca28-1c75-46da-87fd-5941c9ff7eb8_characterizeImage_1124831_50.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/c217ca28-1c75-46da-87fd-5941c9ff7eb8_characterizeImage_1124831_50.stderr parsl.dataflow.dflow INFO: Standard output for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7e53e051-f319-4ffd-a4b5-ee49521cc43b_isr_1124831_50.stdout parsl.dataflow.dflow INFO: Standard error for task 510 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7e53e051-f319-4ffd-a4b5-ee49521cc43b_isr_1124831_50.stderr parsl.dataflow.dflow INFO: Task 474 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8444714c030d45c3a221e3bbf1258f8 with result from task 474 parsl.dataflow.dflow DEBUG: Task 474 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 475 has memoization hash 15fd7aaa28f7e7b42e16138f4144c9ef parsl.dataflow.memoization INFO: Task 475 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981fe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 475 try 0 launched on executor multi with executor id 466 parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/38770e46-f90a-4e56-a26c-0792549e23bb_characterizeImage_1124707_11.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/38770e46-f90a-4e56-a26c-0792549e23bb_characterizeImage_1124707_11.stderr parsl.dataflow.dflow INFO: Standard output for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/7fee85d2-77ec-4396-bb8c-5c0f4901cf1f_isr_1124707_11.stdout parsl.dataflow.dflow INFO: Standard error for task 474 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/7fee85d2-77ec-4396-bb8c-5c0f4901cf1f_isr_1124707_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:03 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 495 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d4e02ba6726739ce7074a7b6d3ad7dd0 with result from task 495 parsl.dataflow.dflow DEBUG: Task 495 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 496 has memoization hash 6de811eeda0f767c14f8594378b63898 parsl.dataflow.memoization INFO: Task 496 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298519e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 496 try 0 launched on executor multi with executor id 467 parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/2de646cb-5a3b-4eb1-b42f-1fec41fbd3ef_characterizeImage_1124707_40.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/2de646cb-5a3b-4eb1-b42f-1fec41fbd3ef_characterizeImage_1124707_40.stderr parsl.dataflow.dflow INFO: Standard output for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c0bd55ef-057b-458f-8f55-c8b125f26f59_isr_1124707_40.stdout parsl.dataflow.dflow INFO: Standard error for task 495 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c0bd55ef-057b-458f-8f55-c8b125f26f59_isr_1124707_40.stderr parsl.dataflow.dflow INFO: Task 507 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49a9fdd1979e49de45fd56c7586edfe1 with result from task 507 parsl.dataflow.dflow DEBUG: Task 507 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 508 has memoization hash 6dac1d57653fb914574b6e974e4abe54 parsl.dataflow.memoization INFO: Task 508 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981efc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 508 try 0 launched on executor multi with executor id 468 parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/9c9b4813-8367-445d-a76f-b74d85dd2992_characterizeImage_1124704_3.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/9c9b4813-8367-445d-a76f-b74d85dd2992_characterizeImage_1124704_3.stderr parsl.dataflow.dflow INFO: Standard output for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/3858304d-3351-4beb-a153-3b9cd12eacca_isr_1124704_3.stdout parsl.dataflow.dflow INFO: Standard error for task 507 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/3858304d-3351-4beb-a153-3b9cd12eacca_isr_1124704_3.stderr parsl.dataflow.dflow INFO: Task 453 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e4da962dc9d465cd37299e3182a3c52 with result from task 453 parsl.dataflow.dflow DEBUG: Task 453 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 454 has memoization hash f66072f11fcb6efe967d7ab286886359 parsl.dataflow.memoization INFO: Task 454 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 454 try 0 launched on executor multi with executor id 469 parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/18550bea-4988-4e18-8264-5e84181b03e5_characterizeImage_1124710_12.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/18550bea-4988-4e18-8264-5e84181b03e5_characterizeImage_1124710_12.stderr parsl.dataflow.dflow INFO: Standard output for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/72b8c9f4-802d-405e-8c45-c652cbe180a6_isr_1124710_12.stdout parsl.dataflow.dflow INFO: Standard error for task 453 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/72b8c9f4-802d-405e-8c45-c652cbe180a6_isr_1124710_12.stderr parsl.dataflow.dflow INFO: Task 462 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2326872e9f9abc010cc103a59693c4a3 with result from task 462 parsl.dataflow.dflow DEBUG: Task 462 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 463 has memoization hash bb4c2b58007ffe57d8285e00fdb17cca parsl.dataflow.memoization INFO: Task 463 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 463 try 0 launched on executor multi with executor id 470 parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/bb2bef86-156e-4b6e-be1a-94a3f7deb1ea_characterizeImage_1124831_46.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/bb2bef86-156e-4b6e-be1a-94a3f7deb1ea_characterizeImage_1124831_46.stderr parsl.dataflow.dflow INFO: Standard output for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/8134fbd9-ddf8-43ed-98d4-cb4f306a5c49_isr_1124831_46.stdout parsl.dataflow.dflow INFO: Standard error for task 462 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/8134fbd9-ddf8-43ed-98d4-cb4f306a5c49_isr_1124831_46.stderr parsl.dataflow.dflow INFO: Task 465 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93a740181253dbd75f01031af89e5042 with result from task 465 parsl.dataflow.dflow DEBUG: Task 465 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 466 has memoization hash 24435e1f5300bdbb6dc16e1ddae5358d parsl.dataflow.memoization INFO: Task 466 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 466 try 0 launched on executor multi with executor id 471 parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/617bf134-50b1-4661-80ee-2ce607a16ca5_characterizeImage_1124831_56.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/617bf134-50b1-4661-80ee-2ce607a16ca5_characterizeImage_1124831_56.stderr parsl.dataflow.dflow INFO: Standard output for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7071c0ff-8b8d-43b6-a3a4-380fc447f612_isr_1124831_56.stdout parsl.dataflow.dflow INFO: Standard error for task 465 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7071c0ff-8b8d-43b6-a3a4-380fc447f612_isr_1124831_56.stderr parsl.dataflow.dflow INFO: Task 477 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 39f24eac014013c0eba3b68a74e4d79f with result from task 477 parsl.dataflow.dflow DEBUG: Task 477 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 478 has memoization hash ab1647b174fe3c0378664f114e841106 parsl.dataflow.memoization INFO: Task 478 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529851d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 478 try 0 launched on executor multi with executor id 472 parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d1cd61fb-2ff9-4ed6-9157-3e2306a7a139_characterizeImage_1124704_11.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d1cd61fb-2ff9-4ed6-9157-3e2306a7a139_characterizeImage_1124704_11.stderr parsl.dataflow.dflow INFO: Standard output for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/683e865c-cbf2-4926-821c-d0fd53d94aa6_isr_1124704_11.stdout parsl.dataflow.dflow INFO: Standard error for task 477 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/683e865c-cbf2-4926-821c-d0fd53d94aa6_isr_1124704_11.stderr parsl.dataflow.dflow INFO: Task 471 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6eb7e821f926b9cdaa5e0fb7a860b1c9 with result from task 471 parsl.dataflow.dflow DEBUG: Task 471 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 472 has memoization hash 0e02f6694082053c0d2c48eb9dc3ab2e parsl.dataflow.memoization INFO: Task 472 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 472 try 0 launched on executor multi with executor id 473 parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/70359c08-b1e9-4bea-b5e9-0b93ed4bd82f_characterizeImage_1124710_47.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/70359c08-b1e9-4bea-b5e9-0b93ed4bd82f_characterizeImage_1124710_47.stderr parsl.dataflow.dflow INFO: Standard output for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/900eb175-97e0-465d-9bce-043b0336b0db_isr_1124710_47.stdout parsl.dataflow.dflow INFO: Standard error for task 471 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/900eb175-97e0-465d-9bce-043b0336b0db_isr_1124710_47.stderr parsl.dataflow.dflow INFO: Task 516 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d2e3f24680b18a343af4d80c84c50d7 with result from task 516 parsl.dataflow.dflow DEBUG: Task 516 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 517 has memoization hash 09dc2e7e8196d00430d9c9aaa697d052 parsl.dataflow.memoization INFO: Task 517 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 517 try 0 launched on executor multi with executor id 474 parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/3af21e47-c9d6-470d-bac3-36f3ee8b128a_characterizeImage_1124831_24.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/3af21e47-c9d6-470d-bac3-36f3ee8b128a_characterizeImage_1124831_24.stderr parsl.dataflow.dflow INFO: Standard output for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/55b72295-7b64-4fca-8e4c-86036301bde2_isr_1124831_24.stdout parsl.dataflow.dflow INFO: Standard error for task 516 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/55b72295-7b64-4fca-8e4c-86036301bde2_isr_1124831_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:04 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 546 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a91ddec92762f91efe9fb679337de558 with result from task 546 parsl.dataflow.dflow DEBUG: Task 546 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 547 has memoization hash 2040c3158e110b96c14e00df4a68f9c1 parsl.dataflow.memoization INFO: Task 547 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 547 try 0 launched on executor multi with executor id 475 parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/45df5da3-3831-4f4a-9681-7a99eb61dc4f_characterizeImage_1124710_4.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/45df5da3-3831-4f4a-9681-7a99eb61dc4f_characterizeImage_1124710_4.stderr parsl.dataflow.dflow INFO: Standard output for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1af31943-db04-4be3-b345-e39f7af1fd63_isr_1124710_4.stdout parsl.dataflow.dflow INFO: Standard error for task 546 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1af31943-db04-4be3-b345-e39f7af1fd63_isr_1124710_4.stderr parsl.dataflow.dflow INFO: Task 468 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2aa09b8e58040f98c6f1b6ee5874544d with result from task 468 parsl.dataflow.dflow DEBUG: Task 468 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 469 has memoization hash 9b240dc26005ed7a8f5f751b2c197751 parsl.dataflow.memoization INFO: Task 469 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981fba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 469 try 0 launched on executor multi with executor id 476 parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f61880eb-cb2c-4b10-9338-b281206dae43_characterizeImage_1124831_54.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f61880eb-cb2c-4b10-9338-b281206dae43_characterizeImage_1124831_54.stderr parsl.dataflow.dflow INFO: Standard output for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/410e1a24-2665-4d29-89bc-67cbd7003a8b_isr_1124831_54.stdout parsl.dataflow.dflow INFO: Standard error for task 468 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/410e1a24-2665-4d29-89bc-67cbd7003a8b_isr_1124831_54.stderr parsl.dataflow.dflow INFO: Task 339 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1da921fd3c2c8f39a096648c86c03f2f with result from task 339 parsl.dataflow.dflow DEBUG: Task 339 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 340 has memoization hash e9dbbfab1456e45e952bda3ce2a8b6aa parsl.dataflow.memoization INFO: Task 340 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cf560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 340 try 0 launched on executor multi with executor id 477 parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/653d08f7-25f6-4f36-90c6-84f4f4a0ac90_characterizeImage_1124707_34.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/653d08f7-25f6-4f36-90c6-84f4f4a0ac90_characterizeImage_1124707_34.stderr parsl.dataflow.dflow INFO: Standard output for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c6cb0395-fbd2-49c3-835b-d9550b64ac83_isr_1124707_34.stdout parsl.dataflow.dflow INFO: Standard error for task 339 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c6cb0395-fbd2-49c3-835b-d9550b64ac83_isr_1124707_34.stderr parsl.dataflow.dflow INFO: Task 330 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 85b51c206e49952d65cb86503dd594ae with result from task 330 parsl.dataflow.dflow DEBUG: Task 330 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 331 has memoization hash cf1bec65d1b86f8e15519812ad7a4258 parsl.dataflow.memoization INFO: Task 331 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 331 try 0 launched on executor multi with executor id 478 parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f0cbeb6e-f4a7-439b-a200-b78899380d7b_characterizeImage_1124704_12.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f0cbeb6e-f4a7-439b-a200-b78899380d7b_characterizeImage_1124704_12.stderr parsl.dataflow.dflow INFO: Standard output for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/b419e3c1-e106-4696-a283-4cb8ab611de8_isr_1124704_12.stdout parsl.dataflow.dflow INFO: Standard error for task 330 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/b419e3c1-e106-4696-a283-4cb8ab611de8_isr_1124704_12.stderr parsl.dataflow.dflow INFO: Task 345 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bd5267ac99bdd07f811fa17e9b68f6c with result from task 345 parsl.dataflow.dflow DEBUG: Task 345 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 346 has memoization hash 4b5ad90fa8deb4a235b18420c8e40fa5 parsl.dataflow.memoization INFO: Task 346 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cfba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 346 try 0 launched on executor multi with executor id 479 parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/181bb79e-c255-406a-afd7-b30869848aa9_characterizeImage_1124710_27.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/181bb79e-c255-406a-afd7-b30869848aa9_characterizeImage_1124710_27.stderr parsl.dataflow.dflow INFO: Standard output for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/ee318162-fa33-47b7-a8f9-9c886a4dacdf_isr_1124710_27.stdout parsl.dataflow.dflow INFO: Standard error for task 345 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/ee318162-fa33-47b7-a8f9-9c886a4dacdf_isr_1124710_27.stderr parsl.dataflow.dflow INFO: Task 336 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78ef158ed8446abad50317c6ce798716 with result from task 336 parsl.dataflow.dflow DEBUG: Task 336 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 337 has memoization hash 1259d88bc3d3a35d904d7fff46167b2f parsl.dataflow.memoization INFO: Task 337 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cf240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 337 try 0 launched on executor multi with executor id 480 parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/c927bf96-ef83-4ba7-a073-9605897f12d4_characterizeImage_1124828_31.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/c927bf96-ef83-4ba7-a073-9605897f12d4_characterizeImage_1124828_31.stderr parsl.dataflow.dflow INFO: Standard output for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/7bf17bef-d950-4c74-8b96-21d00264970c_isr_1124828_31.stdout parsl.dataflow.dflow INFO: Standard error for task 336 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/7bf17bef-d950-4c74-8b96-21d00264970c_isr_1124828_31.stderr parsl.dataflow.dflow INFO: Task 324 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f690692f672b024995bec6a09f562c82 with result from task 324 parsl.dataflow.dflow DEBUG: Task 324 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 325 has memoization hash 19b3f05c1d34097de6e06f0ff97f6fb2 parsl.dataflow.memoization INFO: Task 325 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ce5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 325 try 0 launched on executor multi with executor id 481 parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1ad38249-5484-4675-b0d7-ddca04c6d22b_characterizeImage_1124710_15.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1ad38249-5484-4675-b0d7-ddca04c6d22b_characterizeImage_1124710_15.stderr parsl.dataflow.dflow INFO: Standard output for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/149ac961-b2a1-4d5a-840e-b93ebf78cfeb_isr_1124710_15.stdout parsl.dataflow.dflow INFO: Standard error for task 324 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/149ac961-b2a1-4d5a-840e-b93ebf78cfeb_isr_1124710_15.stderr parsl.dataflow.dflow INFO: Task 555 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5ae13842035feeaeecc6c0607e2c6978 with result from task 555 parsl.dataflow.dflow DEBUG: Task 555 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 556 has memoization hash 1c19dc903a155901c61283b228ccf799 parsl.dataflow.memoization INFO: Task 556 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969d260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 556 try 0 launched on executor multi with executor id 482 parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d5c23c5c-a7d1-41aa-b7d8-1366130690cb_characterizeImage_1124704_17.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d5c23c5c-a7d1-41aa-b7d8-1366130690cb_characterizeImage_1124704_17.stderr parsl.dataflow.dflow INFO: Standard output for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/4e78c99f-11fe-497a-86dd-a12acfc0d8bf_isr_1124704_17.stdout parsl.dataflow.dflow INFO: Standard error for task 555 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/4e78c99f-11fe-497a-86dd-a12acfc0d8bf_isr_1124704_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:05 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 342 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fc75d06b67a524381ea644bc83ab31f with result from task 342 parsl.dataflow.dflow DEBUG: Task 342 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 343 has memoization hash 306fb6e3289a5dd11b8d1ed295a94ed6 parsl.dataflow.memoization INFO: Task 343 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cf880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 343 try 0 launched on executor multi with executor id 483 parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/d0512725-cbda-42fe-bf92-4091d80f1a1c_characterizeImage_1124707_38.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/d0512725-cbda-42fe-bf92-4091d80f1a1c_characterizeImage_1124707_38.stderr parsl.dataflow.dflow INFO: Standard output for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/b4434d0a-a53e-4d09-820e-f8f2070faa58_isr_1124707_38.stdout parsl.dataflow.dflow INFO: Standard error for task 342 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/b4434d0a-a53e-4d09-820e-f8f2070faa58_isr_1124707_38.stderr parsl.dataflow.dflow INFO: Task 591 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a9b11b217ab570486f8f75f86c71234 with result from task 591 parsl.dataflow.dflow DEBUG: Task 591 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash fecb4c03f58e9e88baba6e363c229ef3 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 0 launched on executor multi with executor id 484 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3464ff69-e292-4267-b1de-41e6fa27387c_characterizeImage_1124707_2.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3464ff69-e292-4267-b1de-41e6fa27387c_characterizeImage_1124707_2.stderr parsl.dataflow.dflow INFO: Standard output for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/9e07c330-dc1e-4f28-8b5f-36a4e2702b6e_isr_1124707_2.stdout parsl.dataflow.dflow INFO: Standard error for task 591 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/9e07c330-dc1e-4f28-8b5f-36a4e2702b6e_isr_1124707_2.stderr parsl.dataflow.dflow INFO: Task 327 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c3b990ffdbae059bcc372913fb487e4 with result from task 327 parsl.dataflow.dflow DEBUG: Task 327 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 328 has memoization hash 38d834b36725c58f3da889e1eedaf116 parsl.dataflow.memoization INFO: Task 328 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ce8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 328 try 0 launched on executor multi with executor id 485 parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/cdee9122-7d3c-44df-aefa-f3e2b7642429_characterizeImage_1124828_60.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/cdee9122-7d3c-44df-aefa-f3e2b7642429_characterizeImage_1124828_60.stderr parsl.dataflow.dflow INFO: Standard output for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/41401686-4a91-4f0d-94cb-e4c7fa4fea4c_isr_1124828_60.stdout parsl.dataflow.dflow INFO: Standard error for task 327 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/41401686-4a91-4f0d-94cb-e4c7fa4fea4c_isr_1124828_60.stderr parsl.dataflow.dflow INFO: Task 333 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0f4ec15b644b3f552c1e969fc8079bd with result from task 333 parsl.dataflow.dflow DEBUG: Task 333 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 334 has memoization hash 8036c21b5e2f339721ae3882a2b6b864 parsl.dataflow.memoization INFO: Task 334 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 334 try 0 launched on executor multi with executor id 486 parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/e47b1e95-cbae-4893-919b-2ade846dcd70_characterizeImage_1124831_43.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/e47b1e95-cbae-4893-919b-2ade846dcd70_characterizeImage_1124831_43.stderr parsl.dataflow.dflow INFO: Standard output for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/aa33f37f-ea1e-416b-b7fe-935f87690744_isr_1124831_43.stdout parsl.dataflow.dflow INFO: Standard error for task 333 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/aa33f37f-ea1e-416b-b7fe-935f87690744_isr_1124831_43.stderr parsl.dataflow.dflow INFO: Task 537 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb09180fd244159b7faf24a558cbbef5 with result from task 537 parsl.dataflow.dflow DEBUG: Task 537 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 538 has memoization hash 395a9a4394dade268936568c3e172384 parsl.dataflow.memoization INFO: Task 538 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 538 try 0 launched on executor multi with executor id 487 parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3c89eaff-75e8-4229-84f2-a14aec595db7_characterizeImage_1124707_31.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3c89eaff-75e8-4229-84f2-a14aec595db7_characterizeImage_1124707_31.stderr parsl.dataflow.dflow INFO: Standard output for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/7dd647cf-219b-4b8e-b775-c55064c897df_isr_1124707_31.stdout parsl.dataflow.dflow INFO: Standard error for task 537 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/7dd647cf-219b-4b8e-b775-c55064c897df_isr_1124707_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:06 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 561 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d764a3613366b2042a47cf17150cb727 with result from task 561 parsl.dataflow.dflow DEBUG: Task 561 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 562 has memoization hash ee47230098b5aaa31b8cac8033663955 parsl.dataflow.memoization INFO: Task 562 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 562 try 0 launched on executor multi with executor id 488 parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/b691a32b-f8da-4962-aa3d-c36928ab8193_characterizeImage_1124707_26.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/b691a32b-f8da-4962-aa3d-c36928ab8193_characterizeImage_1124707_26.stderr parsl.dataflow.dflow INFO: Standard output for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/44546abd-d3f5-4ec1-8e40-331c5b87be9a_isr_1124707_26.stdout parsl.dataflow.dflow INFO: Standard error for task 561 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/44546abd-d3f5-4ec1-8e40-331c5b87be9a_isr_1124707_26.stderr parsl.dataflow.dflow INFO: Task 528 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e1f6bc9c17ddda07845e31bf01c1651 with result from task 528 parsl.dataflow.dflow DEBUG: Task 528 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 529 has memoization hash 24058e00161baaad6ae478514cbab109 parsl.dataflow.memoization INFO: Task 529 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529853880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 529 try 0 launched on executor multi with executor id 489 parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/9eb8a533-5fdd-48a4-82f8-18e43206a0cc_characterizeImage_1124828_50.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/9eb8a533-5fdd-48a4-82f8-18e43206a0cc_characterizeImage_1124828_50.stderr parsl.dataflow.dflow INFO: Standard output for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/133eb39e-8dd0-4f8e-a1fc-8388228a6fee_isr_1124828_50.stdout parsl.dataflow.dflow INFO: Standard error for task 528 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/133eb39e-8dd0-4f8e-a1fc-8388228a6fee_isr_1124828_50.stderr parsl.dataflow.dflow INFO: Task 570 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad45d002290cd4080b5c8318a67a9be7 with result from task 570 parsl.dataflow.dflow DEBUG: Task 570 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 571 has memoization hash 6a5c3f81ed3d5a168d8962acd523bb62 parsl.dataflow.memoization INFO: Task 571 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 571 try 0 launched on executor multi with executor id 490 parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/6d9b71bd-b72a-4334-aff2-8035b4b1d808_characterizeImage_1124831_22.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/6d9b71bd-b72a-4334-aff2-8035b4b1d808_characterizeImage_1124831_22.stderr parsl.dataflow.dflow INFO: Standard output for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/830a774e-fe45-47b2-bea5-165fee4a7f06_isr_1124831_22.stdout parsl.dataflow.dflow INFO: Standard error for task 570 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/830a774e-fe45-47b2-bea5-165fee4a7f06_isr_1124831_22.stderr parsl.dataflow.dflow INFO: Task 540 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f1380623dfe389922fe450b25764eb7 with result from task 540 parsl.dataflow.dflow DEBUG: Task 540 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 541 has memoization hash 29d8a1722629c00c0e38bb4a88b26b97 parsl.dataflow.memoization INFO: Task 541 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 541 try 0 launched on executor multi with executor id 491 parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/77026af8-5bad-4205-9629-e4ff84033216_characterizeImage_1124704_33.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/77026af8-5bad-4205-9629-e4ff84033216_characterizeImage_1124704_33.stderr parsl.dataflow.dflow INFO: Standard output for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/42c6a149-5575-4d55-bb65-06e035a599be_isr_1124704_33.stdout parsl.dataflow.dflow INFO: Standard error for task 540 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/42c6a149-5575-4d55-bb65-06e035a599be_isr_1124704_33.stderr parsl.dataflow.dflow INFO: Task 513 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91d0db657723b82e4971dd13e88dcc42 with result from task 513 parsl.dataflow.dflow DEBUG: Task 513 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 514 has memoization hash d0f166ed8f11feb9a4410ea66e505709 parsl.dataflow.memoization INFO: Task 514 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852de0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 514 try 0 launched on executor multi with executor id 492 parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f4852dfc-b0ee-465e-beed-7f80c02ecc57_characterizeImage_1124710_62.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f4852dfc-b0ee-465e-beed-7f80c02ecc57_characterizeImage_1124710_62.stderr parsl.dataflow.dflow INFO: Standard output for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/217cb1bc-ee61-4834-bcf3-8a0ea153d299_isr_1124710_62.stdout parsl.dataflow.dflow INFO: Standard error for task 513 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/217cb1bc-ee61-4834-bcf3-8a0ea153d299_isr_1124710_62.stderr parsl.dataflow.dflow INFO: Task 588 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 596194199623e189c178d42bcf5705ad with result from task 588 parsl.dataflow.dflow DEBUG: Task 588 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 589 has memoization hash 0461687baec30768f4fca159e307850f parsl.dataflow.memoization INFO: Task 589 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 589 try 0 launched on executor multi with executor id 493 parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b6137e75-60d3-4244-b81c-18c00175ae5d_characterizeImage_1124704_35.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b6137e75-60d3-4244-b81c-18c00175ae5d_characterizeImage_1124704_35.stderr parsl.dataflow.dflow INFO: Standard output for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/19b95d54-815a-4717-bbb8-7ddbbfeff40f_isr_1124704_35.stdout parsl.dataflow.dflow INFO: Standard error for task 588 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/19b95d54-815a-4717-bbb8-7ddbbfeff40f_isr_1124704_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 PD 16887776 PD 16887777 PD 16887778 PD 16887779 PD 16887780 PD 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:07 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 522 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffbac31b8e542f7733c942fee7e3a1ad with result from task 522 parsl.dataflow.dflow DEBUG: Task 522 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 523 has memoization hash c1f91212833f56cad9632761f4e1d3d8 parsl.dataflow.memoization INFO: Task 523 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529853100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 523 try 0 launched on executor multi with executor id 494 parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/cd284b62-d70c-4c13-9451-3f60634717ba_characterizeImage_1124831_51.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/cd284b62-d70c-4c13-9451-3f60634717ba_characterizeImage_1124831_51.stderr parsl.dataflow.dflow INFO: Standard output for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/8cb735c4-db77-4508-bf76-2acbd33d7bf5_isr_1124831_51.stdout parsl.dataflow.dflow INFO: Standard error for task 522 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/8cb735c4-db77-4508-bf76-2acbd33d7bf5_isr_1124831_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:08 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 534 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d57cc497b5e9039cc299e8fe8f7a3771 with result from task 534 parsl.dataflow.dflow DEBUG: Task 534 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 535 has memoization hash 20706a80c48bca708dbeb27289b9d5d0 parsl.dataflow.memoization INFO: Task 535 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529853e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 535 try 0 launched on executor multi with executor id 495 parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/9ba0b0b7-31a3-4fa1-8023-f1b0553970cd_characterizeImage_1124707_12.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/9ba0b0b7-31a3-4fa1-8023-f1b0553970cd_characterizeImage_1124707_12.stderr parsl.dataflow.dflow INFO: Standard output for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/1448a12f-91f2-4e44-bafc-e195593a6a46_isr_1124707_12.stdout parsl.dataflow.dflow INFO: Standard error for task 534 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/1448a12f-91f2-4e44-bafc-e195593a6a46_isr_1124707_12.stderr parsl.dataflow.dflow INFO: Task 531 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f22c37b4c4288365b628c708b455d773 with result from task 531 parsl.dataflow.dflow DEBUG: Task 531 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 532 has memoization hash 6bc324dd48ae0246c725307fe8c315c7 parsl.dataflow.memoization INFO: Task 532 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529853d80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 532 try 0 launched on executor multi with executor id 496 parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/42a53e32-44da-498c-95a9-ac2dd1c4f9a3_characterizeImage_1124710_35.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/42a53e32-44da-498c-95a9-ac2dd1c4f9a3_characterizeImage_1124710_35.stderr parsl.dataflow.dflow INFO: Standard output for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/068840a4-e57c-43a7-9922-d7d1d29ec731_isr_1124710_35.stdout parsl.dataflow.dflow INFO: Standard error for task 531 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/068840a4-e57c-43a7-9922-d7d1d29ec731_isr_1124710_35.stderr parsl.dataflow.dflow INFO: Task 525 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e14af678da9eaa094549b20b7514959e with result from task 525 parsl.dataflow.dflow DEBUG: Task 525 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 526 has memoization hash 05672be1ecefe3919c0802c856b43af6 parsl.dataflow.memoization INFO: Task 526 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 526 try 0 launched on executor multi with executor id 497 parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/72c3ea49-7faa-4871-a35f-f4e9b7b62d8b_characterizeImage_1124710_42.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/72c3ea49-7faa-4871-a35f-f4e9b7b62d8b_characterizeImage_1124710_42.stderr parsl.dataflow.dflow INFO: Standard output for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a10765d2-cd77-4a85-9696-51fcd061300e_isr_1124710_42.stdout parsl.dataflow.dflow INFO: Standard error for task 525 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a10765d2-cd77-4a85-9696-51fcd061300e_isr_1124710_42.stderr parsl.dataflow.dflow INFO: Task 543 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f5100e0c6ac4d27fe549f63104758b0 with result from task 543 parsl.dataflow.dflow DEBUG: Task 543 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 544 has memoization hash 26f7b51194ca5ba2638c446a84134ca0 parsl.dataflow.memoization INFO: Task 544 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969c360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 544 try 0 launched on executor multi with executor id 498 parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/51583bb4-d4e7-4552-a6cc-159a431dfaab_characterizeImage_1124710_29.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/51583bb4-d4e7-4552-a6cc-159a431dfaab_characterizeImage_1124710_29.stderr parsl.dataflow.dflow INFO: Standard output for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/b918198a-2942-409a-9e1e-fa0e4821e77e_isr_1124710_29.stdout parsl.dataflow.dflow INFO: Standard error for task 543 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/b918198a-2942-409a-9e1e-fa0e4821e77e_isr_1124710_29.stderr parsl.dataflow.dflow INFO: Task 558 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e8aca9db8cfc5c82fb556e93c53b0ab8 with result from task 558 parsl.dataflow.dflow DEBUG: Task 558 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 559 has memoization hash 4dfbf2a8e1793fd2c2540f011496bb06 parsl.dataflow.memoization INFO: Task 559 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 559 try 0 launched on executor multi with executor id 499 parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5d56936e-b801-4ebc-9a00-4d05a0ce7b15_characterizeImage_1124831_9.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5d56936e-b801-4ebc-9a00-4d05a0ce7b15_characterizeImage_1124831_9.stderr parsl.dataflow.dflow INFO: Standard output for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/3460dbcc-a284-44be-b6bf-ddaf3866d6b2_isr_1124831_9.stdout parsl.dataflow.dflow INFO: Standard error for task 558 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/3460dbcc-a284-44be-b6bf-ddaf3866d6b2_isr_1124831_9.stderr parsl.dataflow.dflow INFO: Task 585 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7167a05ef1c3a9c75d78282b8ee72c57 with result from task 585 parsl.dataflow.dflow DEBUG: Task 585 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 586 has memoization hash 92e5fee8042d7483ae63c3e0a5f2ffb7 parsl.dataflow.memoization INFO: Task 586 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969f880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 586 try 0 launched on executor multi with executor id 500 parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/c2d56486-e16a-4f8d-9186-2fb8c2115d91_characterizeImage_1124707_21.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/c2d56486-e16a-4f8d-9186-2fb8c2115d91_characterizeImage_1124707_21.stderr parsl.dataflow.dflow INFO: Standard output for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c5c78fcd-622a-4f63-a3a1-e85ec445c980_isr_1124707_21.stdout parsl.dataflow.dflow INFO: Standard error for task 585 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c5c78fcd-622a-4f63-a3a1-e85ec445c980_isr_1124707_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:09 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:09 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 597 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd82a5eabe55a5f59a1535c8ef52ec61 with result from task 597 parsl.dataflow.dflow DEBUG: Task 597 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 598 has memoization hash eb3b03c0e9d0a47f383bf85fc2db66cc parsl.dataflow.memoization INFO: Task 598 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e8040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 598 try 0 launched on executor multi with executor id 501 parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/e36ff1ff-ac1f-439b-960f-b10930bc622b_characterizeImage_1124704_1.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/e36ff1ff-ac1f-439b-960f-b10930bc622b_characterizeImage_1124704_1.stderr parsl.dataflow.dflow INFO: Standard output for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a9e8ea36-75be-4dc7-a85d-36f264bfae8a_isr_1124704_1.stdout parsl.dataflow.dflow INFO: Standard error for task 597 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a9e8ea36-75be-4dc7-a85d-36f264bfae8a_isr_1124704_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:10 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 552 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 645ac3365da7757f3db5c6e434148025 with result from task 552 parsl.dataflow.dflow DEBUG: Task 552 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 553 has memoization hash 2aee5d75184e117b18f0f20a422dea06 parsl.dataflow.memoization INFO: Task 553 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 553 try 0 launched on executor multi with executor id 502 parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/53362882-773d-4665-847d-5b99f7f01a17_characterizeImage_1124831_8.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/53362882-773d-4665-847d-5b99f7f01a17_characterizeImage_1124831_8.stderr parsl.dataflow.dflow INFO: Standard output for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/221ade2c-9a3e-4325-bc54-31b66bc2bb6d_isr_1124831_8.stdout parsl.dataflow.dflow INFO: Standard error for task 552 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/221ade2c-9a3e-4325-bc54-31b66bc2bb6d_isr_1124831_8.stderr parsl.dataflow.dflow INFO: Task 579 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ca473b9bcbccf27d2e00463958fc3ff with result from task 579 parsl.dataflow.dflow DEBUG: Task 579 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 580 has memoization hash e741c3ebe598a30d4db4b3ff0849814d parsl.dataflow.memoization INFO: Task 580 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298534c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 580 try 0 launched on executor multi with executor id 503 parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/55201bb7-0888-4988-97f2-a12092ccd529_characterizeImage_1124707_32.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/55201bb7-0888-4988-97f2-a12092ccd529_characterizeImage_1124707_32.stderr parsl.dataflow.dflow INFO: Standard output for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/29f42147-eeb4-47cc-ac6c-d2afc5f49659_isr_1124707_32.stdout parsl.dataflow.dflow INFO: Standard error for task 579 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/29f42147-eeb4-47cc-ac6c-d2afc5f49659_isr_1124707_32.stderr parsl.dataflow.dflow INFO: Task 564 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bb3981b8651653807e20010c90c7dc8 with result from task 564 parsl.dataflow.dflow DEBUG: Task 564 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 565 has memoization hash 9bf1d1a042876a0a7ff8ee6fe885af02 parsl.dataflow.memoization INFO: Task 565 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 565 try 0 launched on executor multi with executor id 504 parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/8b9386b3-1f02-4349-b97f-fc88ead6eaae_characterizeImage_1124828_33.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/8b9386b3-1f02-4349-b97f-fc88ead6eaae_characterizeImage_1124828_33.stderr parsl.dataflow.dflow INFO: Standard output for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/0e85c742-38c5-484a-a504-af0faae07a60_isr_1124828_33.stdout parsl.dataflow.dflow INFO: Standard error for task 564 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/0e85c742-38c5-484a-a504-af0faae07a60_isr_1124828_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:10 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 567 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 278810c18211f830738811af24bbaf19 with result from task 567 parsl.dataflow.dflow DEBUG: Task 567 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 568 has memoization hash c37715cd66faf585f3f3aa9a5c4263e0 parsl.dataflow.memoization INFO: Task 568 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 568 try 0 launched on executor multi with executor id 505 parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f6057aac-61be-4ce6-ab8d-4d8dee8140fa_characterizeImage_1124831_37.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f6057aac-61be-4ce6-ab8d-4d8dee8140fa_characterizeImage_1124831_37.stderr parsl.dataflow.dflow INFO: Standard output for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/1c99e26d-ddad-4b8e-b4c3-7a411fa1483d_isr_1124831_37.stdout parsl.dataflow.dflow INFO: Standard error for task 567 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/1c99e26d-ddad-4b8e-b4c3-7a411fa1483d_isr_1124831_37.stderr parsl.dataflow.dflow INFO: Task 603 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0e7b0abfdb66a513aebe39d58bf391e with result from task 603 parsl.dataflow.dflow DEBUG: Task 603 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 604 has memoization hash ef7b1b831a845eaeed5869092bd658f4 parsl.dataflow.memoization INFO: Task 604 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 604 try 0 launched on executor multi with executor id 506 parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e8e98071-9bbc-48ae-8e22-ad5ae624732d_characterizeImage_1124710_54.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e8e98071-9bbc-48ae-8e22-ad5ae624732d_characterizeImage_1124710_54.stderr parsl.dataflow.dflow INFO: Standard output for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/351d1f22-2310-42c7-9863-c0419fc0c6a8_isr_1124710_54.stdout parsl.dataflow.dflow INFO: Standard error for task 603 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/351d1f22-2310-42c7-9863-c0419fc0c6a8_isr_1124710_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:11 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 573 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c87ec30508469c74f96a5a848ece585 with result from task 573 parsl.dataflow.dflow DEBUG: Task 573 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 574 has memoization hash 317a958e68b41741aef10699e96a7e89 parsl.dataflow.memoization INFO: Task 574 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 574 try 0 launched on executor multi with executor id 507 parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/6c92ec33-3131-4263-a109-38303ba77a06_characterizeImage_1124704_42.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/6c92ec33-3131-4263-a109-38303ba77a06_characterizeImage_1124704_42.stderr parsl.dataflow.dflow INFO: Standard output for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/c23fdd69-5589-4cfe-893e-e839d082b4a2_isr_1124704_42.stdout parsl.dataflow.dflow INFO: Standard error for task 573 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/c23fdd69-5589-4cfe-893e-e839d082b4a2_isr_1124704_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:12 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 576 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6081c5e8f580ffab56231aff98efb278 with result from task 576 parsl.dataflow.dflow DEBUG: Task 576 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 577 has memoization hash 0c0169a60f4292fb6dfe4e09defac0a8 parsl.dataflow.memoization INFO: Task 577 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 577 try 0 launched on executor multi with executor id 508 parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/d9d76f9e-a9d6-4ce7-8868-57f4d6169e38_characterizeImage_1124831_55.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/d9d76f9e-a9d6-4ce7-8868-57f4d6169e38_characterizeImage_1124831_55.stderr parsl.dataflow.dflow INFO: Standard output for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/feb34aab-6bf5-48b7-8876-2d45ef173ecd_isr_1124831_55.stdout parsl.dataflow.dflow INFO: Standard error for task 576 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/feb34aab-6bf5-48b7-8876-2d45ef173ecd_isr_1124831_55.stderr parsl.dataflow.dflow INFO: Task 549 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12560c3662a6eefdf58763ba855e6dc7 with result from task 549 parsl.dataflow.dflow DEBUG: Task 549 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 550 has memoization hash 68db9990fa1af305e9f35a7d89bcc429 parsl.dataflow.memoization INFO: Task 550 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969cae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 550 try 0 launched on executor multi with executor id 509 parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b754b0a6-5cf5-4fa6-ae04-083027463a19_characterizeImage_1124704_53.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b754b0a6-5cf5-4fa6-ae04-083027463a19_characterizeImage_1124704_53.stderr parsl.dataflow.dflow INFO: Standard output for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/f2fc72bf-44fb-4558-bd6d-55c042ae7986_isr_1124704_53.stdout parsl.dataflow.dflow INFO: Standard error for task 549 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/f2fc72bf-44fb-4558-bd6d-55c042ae7986_isr_1124704_53.stderr parsl.dataflow.dflow INFO: Task 600 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e13886f1b0125790181bf7d4375652e with result from task 600 parsl.dataflow.dflow DEBUG: Task 600 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 601 has memoization hash 0e0c1b00881461c3c006c1b41ee83f1c parsl.dataflow.memoization INFO: Task 601 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 601 try 0 launched on executor multi with executor id 510 parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/48c02441-1d48-4b40-8565-b05cfc80db4c_characterizeImage_1124828_10.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/48c02441-1d48-4b40-8565-b05cfc80db4c_characterizeImage_1124828_10.stderr parsl.dataflow.dflow INFO: Standard output for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/db0cd417-1bfd-4ccf-a80c-b9cb861fce3e_isr_1124828_10.stdout parsl.dataflow.dflow INFO: Standard error for task 600 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/db0cd417-1bfd-4ccf-a80c-b9cb861fce3e_isr_1124828_10.stderr parsl.dataflow.dflow INFO: Task 609 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7822b09914827cce77a0f3f9ca90425 with result from task 609 parsl.dataflow.dflow DEBUG: Task 609 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 610 has memoization hash f9305c11a8e40d2e290c4aca4fc3bd24 parsl.dataflow.memoization INFO: Task 610 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 610 try 0 launched on executor multi with executor id 511 parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/86d61e2d-f73a-459a-8135-9daafc3fb8d4_characterizeImage_1124707_18.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/86d61e2d-f73a-459a-8135-9daafc3fb8d4_characterizeImage_1124707_18.stderr parsl.dataflow.dflow INFO: Standard output for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/75dc2007-9b7e-454c-a953-23dbfa54b278_isr_1124707_18.stdout parsl.dataflow.dflow INFO: Standard error for task 609 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/75dc2007-9b7e-454c-a953-23dbfa54b278_isr_1124707_18.stderr parsl.dataflow.dflow INFO: Task 621 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7eb5cc690ccd97e36b5de8b0659d66d1 with result from task 621 parsl.dataflow.dflow DEBUG: Task 621 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 622 has memoization hash 6aad0f2884cfc62177f90f409f90db3a parsl.dataflow.memoization INFO: Task 622 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 622 try 0 launched on executor multi with executor id 512 parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/62a67ab7-db44-43e4-a14e-3695b34abae4_characterizeImage_1124704_29.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/62a67ab7-db44-43e4-a14e-3695b34abae4_characterizeImage_1124704_29.stderr parsl.dataflow.dflow INFO: Standard output for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/263b5dfa-631c-4d2e-a478-d939197a9b83_isr_1124704_29.stdout parsl.dataflow.dflow INFO: Standard error for task 621 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/263b5dfa-631c-4d2e-a478-d939197a9b83_isr_1124704_29.stderr parsl.dataflow.dflow INFO: Task 582 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 213199a382eca8cf59c2458760831f90 with result from task 582 parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:12 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow DEBUG: Task 582 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 583 has memoization hash 055e6746995c0baa10ae49800fcda7a4 parsl.dataflow.memoization INFO: Task 583 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 583 try 0 launched on executor multi with executor id 513 parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a2a10ae1-5b06-49d9-b9c1-ce1d04c44c55_characterizeImage_1124828_54.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a2a10ae1-5b06-49d9-b9c1-ce1d04c44c55_characterizeImage_1124828_54.stderr parsl.dataflow.dflow INFO: Standard output for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/78c7e4db-12f0-49b9-9f7a-2f5c0f51f6df_isr_1124828_54.stdout parsl.dataflow.dflow INFO: Standard error for task 582 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/78c7e4db-12f0-49b9-9f7a-2f5c0f51f6df_isr_1124828_54.stderr parsl.dataflow.dflow INFO: Task 594 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc2294d5b6e30dc1ec4ddfe1d29eba23 with result from task 594 parsl.dataflow.dflow DEBUG: Task 594 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 595 has memoization hash 89210d7d95b95ed654e710f9b6d1778b parsl.dataflow.memoization INFO: Task 595 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 595 try 0 launched on executor multi with executor id 514 parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/ae38fb96-1070-4562-92d8-5600f2518c05_characterizeImage_1124707_58.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/ae38fb96-1070-4562-92d8-5600f2518c05_characterizeImage_1124707_58.stderr parsl.dataflow.dflow INFO: Standard output for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/f623c370-fd38-4159-953a-56c2d9dcd1d7_isr_1124707_58.stdout parsl.dataflow.dflow INFO: Standard error for task 594 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/f623c370-fd38-4159-953a-56c2d9dcd1d7_isr_1124707_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:13 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 606 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 422eb27efd09a255e8874f0b5457e010 with result from task 606 parsl.dataflow.dflow DEBUG: Task 606 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 607 has memoization hash 03dbf4e6b12f7efe5c32f4a48c63399f parsl.dataflow.memoization INFO: Task 607 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 607 try 0 launched on executor multi with executor id 515 parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/d43d61ce-a68a-4b07-9f31-8cc93c1e43f7_characterizeImage_1124828_1.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/d43d61ce-a68a-4b07-9f31-8cc93c1e43f7_characterizeImage_1124828_1.stderr parsl.dataflow.dflow INFO: Standard output for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/f412f37b-7b9f-4217-a5d8-a278e44853fc_isr_1124828_1.stdout parsl.dataflow.dflow INFO: Standard error for task 606 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/f412f37b-7b9f-4217-a5d8-a278e44853fc_isr_1124828_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:13 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 612 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 819f92182f723f902337444d337d4eee with result from task 612 parsl.dataflow.dflow DEBUG: Task 612 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 613 has memoization hash 5818de60ea699db1e404da198f958abd parsl.dataflow.memoization INFO: Task 613 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 613 try 0 launched on executor multi with executor id 516 parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/2af2eed8-323c-489d-ba5c-36a56e3423ee_characterizeImage_1124710_24.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/2af2eed8-323c-489d-ba5c-36a56e3423ee_characterizeImage_1124710_24.stderr parsl.dataflow.dflow INFO: Standard output for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/dc9a070c-4197-4937-9506-d82a1f385b08_isr_1124710_24.stdout parsl.dataflow.dflow INFO: Standard error for task 612 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/dc9a070c-4197-4937-9506-d82a1f385b08_isr_1124710_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:14 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 639 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 119585d52bf305d4b0ed9b7f9145e2e6 with result from task 639 parsl.dataflow.dflow DEBUG: Task 639 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 640 has memoization hash 3cefb392aeaaf224f0c375273071647a parsl.dataflow.memoization INFO: Task 640 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eb2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 640 try 0 launched on executor multi with executor id 517 parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/23b1db91-d656-4f10-893a-3a46427618c8_characterizeImage_1124707_56.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/23b1db91-d656-4f10-893a-3a46427618c8_characterizeImage_1124707_56.stderr parsl.dataflow.dflow INFO: Standard output for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5c572b92-f773-4afc-9992-8061e5234d30_isr_1124707_56.stdout parsl.dataflow.dflow INFO: Standard error for task 639 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/5c572b92-f773-4afc-9992-8061e5234d30_isr_1124707_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:15 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 615 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c161a01aafdf7f30fb6ed3b8d49f5c0 with result from task 615 parsl.dataflow.dflow DEBUG: Task 615 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 616 has memoization hash c31175cc83f241284902b8cb4dace71f parsl.dataflow.memoization INFO: Task 616 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e91c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 616 try 0 launched on executor multi with executor id 518 parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1232cf25-d352-4464-8cc6-21a6c6bcfb39_characterizeImage_1124710_58.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1232cf25-d352-4464-8cc6-21a6c6bcfb39_characterizeImage_1124710_58.stderr parsl.dataflow.dflow INFO: Standard output for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/3806cec8-ca93-4d5f-ba91-8d964c5e31f1_isr_1124710_58.stdout parsl.dataflow.dflow INFO: Standard error for task 615 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/3806cec8-ca93-4d5f-ba91-8d964c5e31f1_isr_1124710_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:16 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 636 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53bb6b26757cccf86d5b9dbedc4938a2 with result from task 636 parsl.dataflow.dflow DEBUG: Task 636 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 637 has memoization hash d9fdf81ce2f13ad13ba1d7fb2e6daf23 parsl.dataflow.memoization INFO: Task 637 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eafc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 637 try 0 launched on executor multi with executor id 519 parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/93f141fd-3d59-4a8c-be13-47ac066c22ca_characterizeImage_1124707_19.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/93f141fd-3d59-4a8c-be13-47ac066c22ca_characterizeImage_1124707_19.stderr parsl.dataflow.dflow INFO: Standard output for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/d2f9dc24-7e05-4169-a108-9a02f25b69a2_isr_1124707_19.stdout parsl.dataflow.dflow INFO: Standard error for task 636 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/d2f9dc24-7e05-4169-a108-9a02f25b69a2_isr_1124707_19.stderr parsl.dataflow.dflow INFO: Task 654 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5a17f12b74611c871aca7a994c2eae8 with result from task 654 parsl.dataflow.dflow DEBUG: Task 654 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 655 has memoization hash ad776372aeababb4bcdb46c27c806795 parsl.dataflow.memoization INFO: Task 655 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969f600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 655 try 0 launched on executor multi with executor id 520 parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d0c93e16-506f-4a62-9426-066a2b91cffd_characterizeImage_1124710_19.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d0c93e16-506f-4a62-9426-066a2b91cffd_characterizeImage_1124710_19.stderr parsl.dataflow.dflow INFO: Standard output for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f8ae11b2-d715-4bea-92de-09a802de7db6_isr_1124710_19.stdout parsl.dataflow.dflow INFO: Standard error for task 654 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f8ae11b2-d715-4bea-92de-09a802de7db6_isr_1124710_19.stderr parsl.dataflow.dflow INFO: Task 693 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e17c2bfe5d36e21c4fdcfbe5e3f7c5ee with result from task 693 parsl.dataflow.dflow DEBUG: Task 693 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 694 has memoization hash eaeaad8f9f54e8d376c57a760158c850 parsl.dataflow.memoization INFO: Task 694 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973a840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 694 try 0 launched on executor multi with executor id 521 parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0976be9d-81ed-4361-91df-87563f5a3d04_characterizeImage_1124828_25.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0976be9d-81ed-4361-91df-87563f5a3d04_characterizeImage_1124828_25.stderr parsl.dataflow.dflow INFO: Standard output for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/e382886f-7c7a-4045-aab3-ab3aa40c6b51_isr_1124828_25.stdout parsl.dataflow.dflow INFO: Standard error for task 693 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/e382886f-7c7a-4045-aab3-ab3aa40c6b51_isr_1124828_25.stderr parsl.dataflow.dflow INFO: Task 618 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b993d5ed18b3d11215e7b39f734df197 with result from task 618 parsl.dataflow.dflow DEBUG: Task 618 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 619 has memoization hash e3c3c9b60f7b888ffc01b82534b5a6bf parsl.dataflow.memoization INFO: Task 619 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 619 try 0 launched on executor multi with executor id 522 parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/6c917eda-39fa-42ee-af5c-58c6a69d9ce2_characterizeImage_1124828_55.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/6c917eda-39fa-42ee-af5c-58c6a69d9ce2_characterizeImage_1124828_55.stderr parsl.dataflow.dflow INFO: Standard output for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bc8f14dd-de00-4ee3-947a-14e1f9fc6db2_isr_1124828_55.stdout parsl.dataflow.dflow INFO: Standard error for task 618 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bc8f14dd-de00-4ee3-947a-14e1f9fc6db2_isr_1124828_55.stderr parsl.dataflow.dflow INFO: Task 744 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a9d33351b4356543da08e6d2e9cd2df8 with result from task 744 parsl.dataflow.dflow DEBUG: Task 744 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 745 has memoization hash dbadc6fff26178ed3677899f8fdc31a6 parsl.dataflow.memoization INFO: Task 745 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977d8a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 745 try 0 launched on executor multi with executor id 523 parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5ec2fb67-443f-424e-b7bb-32ff6333c218_characterizeImage_1124831_39.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5ec2fb67-443f-424e-b7bb-32ff6333c218_characterizeImage_1124831_39.stderr parsl.dataflow.dflow INFO: Standard output for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/497e5620-0715-414a-bb16-cc05c562fbb8_isr_1124831_39.stdout parsl.dataflow.dflow INFO: Standard error for task 744 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/497e5620-0715-414a-bb16-cc05c562fbb8_isr_1124831_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:17 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 633 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 370c33e9e537851e7310dd13a8200ba6 with result from task 633 parsl.dataflow.dflow DEBUG: Task 633 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 634 has memoization hash 477e5023e7581ab198be3ccba8aa8e84 parsl.dataflow.memoization INFO: Task 634 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 634 try 0 launched on executor multi with executor id 524 parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/6d7a3e7d-54dc-420d-beb2-d4927271ff3b_characterizeImage_1124710_21.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/6d7a3e7d-54dc-420d-beb2-d4927271ff3b_characterizeImage_1124710_21.stderr parsl.dataflow.dflow INFO: Standard output for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/6c654ce8-c47f-4792-bf0d-27b1aeccd86f_isr_1124710_21.stdout parsl.dataflow.dflow INFO: Standard error for task 633 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/6c654ce8-c47f-4792-bf0d-27b1aeccd86f_isr_1124710_21.stderr parsl.dataflow.dflow INFO: Task 738 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8a991d791649d8ae1ebe967396199d6 with result from task 738 parsl.dataflow.dflow DEBUG: Task 738 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 739 has memoization hash 604b3101a249b740bf140b2833239e16 parsl.dataflow.memoization INFO: Task 739 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977d300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 739 try 0 launched on executor multi with executor id 525 parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/7a9dafea-43bb-4cf1-baaa-691b79839bdf_characterizeImage_1124707_35.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/7a9dafea-43bb-4cf1-baaa-691b79839bdf_characterizeImage_1124707_35.stderr parsl.dataflow.dflow INFO: Standard output for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/cf3bfe69-1eaa-49dc-be02-13e74fb8b6fe_isr_1124707_35.stdout parsl.dataflow.dflow INFO: Standard error for task 738 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/cf3bfe69-1eaa-49dc-be02-13e74fb8b6fe_isr_1124707_35.stderr parsl.dataflow.dflow INFO: Task 786 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 28f70f4b086de8531ac8c05ae60b7fa6 with result from task 786 parsl.dataflow.dflow DEBUG: Task 786 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 787 has memoization hash d6deabfeaaf9165ed93012871273a6f8 parsl.dataflow.memoization INFO: Task 787 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 787 try 0 launched on executor multi with executor id 526 parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/93f38e2d-acf6-4dca-90c6-5443e281453c_characterizeImage_1124828_5.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/93f38e2d-acf6-4dca-90c6-5443e281453c_characterizeImage_1124828_5.stderr parsl.dataflow.dflow INFO: Standard output for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bf4f389f-20dc-4c41-a679-3f9dd5e9244f_isr_1124828_5.stdout parsl.dataflow.dflow INFO: Standard error for task 786 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bf4f389f-20dc-4c41-a679-3f9dd5e9244f_isr_1124828_5.stderr parsl.dataflow.dflow INFO: Task 624 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e2ed1fed239e6f969176d32a1911788 with result from task 624 parsl.dataflow.dflow DEBUG: Task 624 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 625 has memoization hash ee3176055ded0617d4eae17e04e35fa7 parsl.dataflow.memoization INFO: Task 625 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ea5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 625 try 0 launched on executor multi with executor id 527 parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/47aa9eb7-f525-484c-9193-1ed9647e81c5_characterizeImage_1124710_45.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/47aa9eb7-f525-484c-9193-1ed9647e81c5_characterizeImage_1124710_45.stderr parsl.dataflow.dflow INFO: Standard output for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/c8c83bc3-706b-4912-a018-0992989d7c46_isr_1124710_45.stdout parsl.dataflow.dflow INFO: Standard error for task 624 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/c8c83bc3-706b-4912-a018-0992989d7c46_isr_1124710_45.stderr parsl.dataflow.dflow INFO: Task 645 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfe92bcc7457f7d5ed808d7eac14e8bd with result from task 645 parsl.dataflow.dflow DEBUG: Task 645 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 646 has memoization hash e7eb81ff02f7cebc92f820e74601210a parsl.dataflow.memoization INFO: Task 646 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eb920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 646 try 0 launched on executor multi with executor id 528 parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7ce38cdd-896b-41f4-8edc-e7baf1da2008_characterizeImage_1124710_1.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7ce38cdd-896b-41f4-8edc-e7baf1da2008_characterizeImage_1124710_1.stderr parsl.dataflow.dflow INFO: Standard output for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1afbedac-bf9f-4918-a17f-e47fc3ef846c_isr_1124710_1.stdout parsl.dataflow.dflow INFO: Standard error for task 645 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/1afbedac-bf9f-4918-a17f-e47fc3ef846c_isr_1124710_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:18 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 732 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bdaea3ce4e60a5b36fa9b1c5b7524fa with result from task 732 parsl.dataflow.dflow DEBUG: Task 732 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 0e6631dc1017587635539c370f059183 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 0 launched on executor multi with executor id 529 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/23f01ed6-d153-43be-8984-62e5f76efe73_characterizeImage_1124710_2.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/23f01ed6-d153-43be-8984-62e5f76efe73_characterizeImage_1124710_2.stderr parsl.dataflow.dflow INFO: Standard output for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/4921318f-d942-42b2-93c2-54761420e06c_isr_1124710_2.stdout parsl.dataflow.dflow INFO: Standard error for task 732 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/4921318f-d942-42b2-93c2-54761420e06c_isr_1124710_2.stderr parsl.dataflow.dflow INFO: Task 663 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e2167023b15e8108a0375ac7f265534 with result from task 663 parsl.dataflow.dflow DEBUG: Task 663 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 664 has memoization hash 16b8f61ba9e1d1f20187aa04eabde6bd parsl.dataflow.memoization INFO: Task 664 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529738a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 664 try 0 launched on executor multi with executor id 530 parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b7ddbbeb-c55b-437f-8db6-108f505fe4be_characterizeImage_1124710_39.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b7ddbbeb-c55b-437f-8db6-108f505fe4be_characterizeImage_1124710_39.stderr parsl.dataflow.dflow INFO: Standard output for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/055cea0c-2a55-45d3-bcfc-76bb629986ca_isr_1124710_39.stdout parsl.dataflow.dflow INFO: Standard error for task 663 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/055cea0c-2a55-45d3-bcfc-76bb629986ca_isr_1124710_39.stderr parsl.dataflow.dflow INFO: Task 627 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24992708581f4b057172751e01719913 with result from task 627 parsl.dataflow.dflow DEBUG: Task 627 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 628 has memoization hash 2e93ea032a900d2a454692dbb06f1a1e parsl.dataflow.memoization INFO: Task 628 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ea2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 628 try 0 launched on executor multi with executor id 531 parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/3f198998-89e0-4665-9e19-64f339f80f51_characterizeImage_1124831_35.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/3f198998-89e0-4665-9e19-64f339f80f51_characterizeImage_1124831_35.stderr parsl.dataflow.dflow INFO: Standard output for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/4504eb1c-f18d-4446-822f-7abc20a8a2c8_isr_1124831_35.stdout parsl.dataflow.dflow INFO: Standard error for task 627 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/4504eb1c-f18d-4446-822f-7abc20a8a2c8_isr_1124831_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:19 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 630 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09b8668041ceef1019ba64f074527ee2 with result from task 630 parsl.dataflow.dflow DEBUG: Task 630 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 631 has memoization hash 8b9cbfdce314c62a097e72b3663b5c96 parsl.dataflow.memoization INFO: Task 631 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e9f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 631 try 0 launched on executor multi with executor id 532 parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/76ab926f-8fa6-4a79-9ca1-1a878448a2b4_characterizeImage_1124828_14.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/76ab926f-8fa6-4a79-9ca1-1a878448a2b4_characterizeImage_1124828_14.stderr parsl.dataflow.dflow INFO: Standard output for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/dd8a025d-27de-43c3-af80-bc0e389106c2_isr_1124828_14.stdout parsl.dataflow.dflow INFO: Standard error for task 630 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/dd8a025d-27de-43c3-af80-bc0e389106c2_isr_1124828_14.stderr parsl.dataflow.dflow INFO: Task 780 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05994e2d182a07693677016f45312a48 with result from task 780 parsl.dataflow.dflow DEBUG: Task 780 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 781 has memoization hash cecf837fd0cb9720206f21ac4545e0f2 parsl.dataflow.memoization INFO: Task 781 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 781 try 0 launched on executor multi with executor id 533 parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/46ac0cf9-d797-4f86-9543-d4970f62c1fc_characterizeImage_1124704_52.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/46ac0cf9-d797-4f86-9543-d4970f62c1fc_characterizeImage_1124704_52.stderr parsl.dataflow.dflow INFO: Standard output for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/5ff1fe47-fda0-4675-955d-4e3e8c8132be_isr_1124704_52.stdout parsl.dataflow.dflow INFO: Standard error for task 780 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/5ff1fe47-fda0-4675-955d-4e3e8c8132be_isr_1124704_52.stderr parsl.dataflow.dflow INFO: Task 687 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1f4f3f1e67d23878212d612b7cb97c2 with result from task 687 parsl.dataflow.dflow DEBUG: Task 687 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 688 has memoization hash c1b873a87a65fa96bf857407baa2f1e7 parsl.dataflow.memoization INFO: Task 688 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973a0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 688 try 0 launched on executor multi with executor id 534 parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/509c9765-30ac-420c-8f1d-fe050208357f_characterizeImage_1124704_21.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/509c9765-30ac-420c-8f1d-fe050208357f_characterizeImage_1124704_21.stderr parsl.dataflow.dflow INFO: Standard output for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e052651e-2c4e-4c43-ab49-2f6d84e55b5e_isr_1124704_21.stdout parsl.dataflow.dflow INFO: Standard error for task 687 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e052651e-2c4e-4c43-ab49-2f6d84e55b5e_isr_1124704_21.stderr parsl.dataflow.dflow INFO: Task 843 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc329a7e991f999525da436c84c711e8 with result from task 843 parsl.dataflow.dflow DEBUG: Task 843 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 844 has memoization hash afe2622ec39bae43ddfb03c5e724cc28 parsl.dataflow.memoization INFO: Task 844 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 844 try 0 launched on executor multi with executor id 535 parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/68c3cbf4-38fa-4953-8554-c5a8cae02a7f_characterizeImage_1124707_62.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/68c3cbf4-38fa-4953-8554-c5a8cae02a7f_characterizeImage_1124707_62.stderr parsl.dataflow.dflow INFO: Standard output for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/ddb1a578-9592-4d55-bda1-ea474f0f2c4e_isr_1124707_62.stdout parsl.dataflow.dflow INFO: Standard error for task 843 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/ddb1a578-9592-4d55-bda1-ea474f0f2c4e_isr_1124707_62.stderr parsl.dataflow.dflow INFO: Task 681 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c705a5609630110653986a9341ec14fa with result from task 681 parsl.dataflow.dflow DEBUG: Task 681 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 682 has memoization hash e45239e11a4b98f59bd21aa3373385de parsl.dataflow.memoization INFO: Task 682 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529739a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 682 try 0 launched on executor multi with executor id 536 parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/aff787f5-9c21-4cd7-84a6-e85110b98875_characterizeImage_1124710_11.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/aff787f5-9c21-4cd7-84a6-e85110b98875_characterizeImage_1124710_11.stderr parsl.dataflow.dflow INFO: Standard output for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/3c349fe0-410a-476c-bd50-265e41e27a1b_isr_1124710_11.stdout parsl.dataflow.dflow INFO: Standard error for task 681 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/3c349fe0-410a-476c-bd50-265e41e27a1b_isr_1124710_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:20 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 690 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e3c7a7d02f236e9778a3cc6b72ee298 with result from task 690 parsl.dataflow.dflow DEBUG: Task 690 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 691 has memoization hash 98313d3abf97c60c7779df7664af418c parsl.dataflow.memoization INFO: Task 691 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973a520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 691 try 0 launched on executor multi with executor id 537 parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/494232c3-befd-423c-a436-084ed2d7c04f_characterizeImage_1124828_53.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/494232c3-befd-423c-a436-084ed2d7c04f_characterizeImage_1124828_53.stderr parsl.dataflow.dflow INFO: Standard output for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/c9895ecc-9fa4-4145-8d16-53c5fec4c357_isr_1124828_53.stdout parsl.dataflow.dflow INFO: Standard error for task 690 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/c9895ecc-9fa4-4145-8d16-53c5fec4c357_isr_1124828_53.stderr parsl.dataflow.dflow INFO: Task 723 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de5f86987175c0dbba17fee4701f49bb with result from task 723 parsl.dataflow.dflow DEBUG: Task 723 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 724 has memoization hash ffe23b9c905fb1fcecfd21486cc4c2a8 parsl.dataflow.memoization INFO: Task 724 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297389a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 724 try 0 launched on executor multi with executor id 538 parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bdb5162a-f9e6-43b6-b996-95f0f68b1710_characterizeImage_1124704_5.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bdb5162a-f9e6-43b6-b996-95f0f68b1710_characterizeImage_1124704_5.stderr parsl.dataflow.dflow INFO: Standard output for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/9a5c3e5e-7b76-4dd0-896f-aac9cd8abf5d_isr_1124704_5.stdout parsl.dataflow.dflow INFO: Standard error for task 723 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/9a5c3e5e-7b76-4dd0-896f-aac9cd8abf5d_isr_1124704_5.stderr parsl.dataflow.dflow INFO: Task 672 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1a35ce72024706e55185640c509ca29 with result from task 672 parsl.dataflow.dflow DEBUG: Task 672 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 673 has memoization hash 7a06dd98207af8afddf6197b88b7eb8f parsl.dataflow.memoization INFO: Task 673 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529739260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 673 try 0 launched on executor multi with executor id 539 parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/958ee1be-e054-4b11-9bb2-a7f9eac43bf3_characterizeImage_1124704_55.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/958ee1be-e054-4b11-9bb2-a7f9eac43bf3_characterizeImage_1124704_55.stderr parsl.dataflow.dflow INFO: Standard output for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/2b3ca996-a46d-4047-bec0-8a615537f328_isr_1124704_55.stdout parsl.dataflow.dflow INFO: Standard error for task 672 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/2b3ca996-a46d-4047-bec0-8a615537f328_isr_1124704_55.stderr parsl.dataflow.dflow INFO: Task 660 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8213ce06368e93d70fec11c8ffe6e8fc with result from task 660 parsl.dataflow.dflow DEBUG: Task 660 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 661 has memoization hash fb7cf0d0b55e390bc1ea1ce7667dfeef parsl.dataflow.memoization INFO: Task 661 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529738720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 661 try 0 launched on executor multi with executor id 540 parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e00ff511-24e2-45fc-91b7-f241c62d1d13_characterizeImage_1124710_49.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e00ff511-24e2-45fc-91b7-f241c62d1d13_characterizeImage_1124710_49.stderr parsl.dataflow.dflow INFO: Standard output for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/15dd493b-3d75-404f-acfa-f49a422efadc_isr_1124710_49.stdout parsl.dataflow.dflow INFO: Standard error for task 660 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/15dd493b-3d75-404f-acfa-f49a422efadc_isr_1124710_49.stderr parsl.dataflow.dflow INFO: Task 642 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d62557e38ee0c7586b7fefe26ae79f80 with result from task 642 parsl.dataflow.dflow DEBUG: Task 642 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 643 has memoization hash 8ea1ef677d3f12b98e9764f9f3f70c0e parsl.dataflow.memoization INFO: Task 643 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:21 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Parsl task 643 try 0 launched on executor multi with executor id 541 parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/b4e25147-cdae-4fa9-bfdd-626a9b1ac245_characterizeImage_1124707_6.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/b4e25147-cdae-4fa9-bfdd-626a9b1ac245_characterizeImage_1124707_6.stderr parsl.dataflow.dflow INFO: Standard output for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/931477e4-ab28-4028-a86f-02b5a22073ca_isr_1124707_6.stdout parsl.dataflow.dflow INFO: Standard error for task 642 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/931477e4-ab28-4028-a86f-02b5a22073ca_isr_1124707_6.stderr parsl.dataflow.dflow INFO: Task 651 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78044617628f5cca4ef97ed20fa5f464 with result from task 651 parsl.dataflow.dflow DEBUG: Task 651 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 652 has memoization hash 2b4accf1940013662856fce70dfcfd35 parsl.dataflow.memoization INFO: Task 652 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ebf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 652 try 0 launched on executor multi with executor id 542 parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/d9e21934-7cee-428c-875c-c1f03aa9727b_characterizeImage_1124707_55.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/d9e21934-7cee-428c-875c-c1f03aa9727b_characterizeImage_1124707_55.stderr parsl.dataflow.dflow INFO: Standard output for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0e24fc76-6e41-447c-8568-019ba9745961_isr_1124707_55.stdout parsl.dataflow.dflow INFO: Standard error for task 651 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0e24fc76-6e41-447c-8568-019ba9745961_isr_1124707_55.stderr parsl.dataflow.dflow INFO: Task 675 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e89d37be8529d1abae5a29d949d8a0e with result from task 675 parsl.dataflow.dflow DEBUG: Task 675 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 676 has memoization hash 1e8962a5e87707f40e60e7f6df2b9c70 parsl.dataflow.memoization INFO: Task 676 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529739580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 676 try 0 launched on executor multi with executor id 543 parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/abcd9725-7e72-4b25-8d67-67809a297596_characterizeImage_1124831_10.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/abcd9725-7e72-4b25-8d67-67809a297596_characterizeImage_1124831_10.stderr parsl.dataflow.dflow INFO: Standard output for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/b516c265-2fb4-4ba4-8e7f-21d9d142b022_isr_1124831_10.stdout parsl.dataflow.dflow INFO: Standard error for task 675 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/b516c265-2fb4-4ba4-8e7f-21d9d142b022_isr_1124831_10.stderr parsl.dataflow.dflow INFO: Task 750 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2543ca82205761a294949e4b4eacc89 with result from task 750 parsl.dataflow.dflow DEBUG: Task 750 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 751 has memoization hash 7649e6606d6f9a14a3d401bf8b780b25 parsl.dataflow.memoization INFO: Task 751 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977df80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 751 try 0 launched on executor multi with executor id 544 parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5b5b3f2c-06e3-4190-9ea9-5504df1e2393_characterizeImage_1124831_11.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5b5b3f2c-06e3-4190-9ea9-5504df1e2393_characterizeImage_1124831_11.stderr parsl.dataflow.dflow INFO: Standard output for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/692bdb39-2274-4e50-a2f1-2c606e8edbab_isr_1124831_11.stdout parsl.dataflow.dflow INFO: Standard error for task 750 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/692bdb39-2274-4e50-a2f1-2c606e8edbab_isr_1124831_11.stderr parsl.dataflow.dflow INFO: Task 696 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5756e925c2c38f9b09c5d7326ad5f01 with result from task 696 parsl.dataflow.dflow DEBUG: Task 696 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 697 has memoization hash 971ab3bce9fe6ad8fb9f2f0987caa812 parsl.dataflow.memoization INFO: Task 697 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 697 try 0 launched on executor multi with executor id 545 parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d1431c49-73d9-41f4-aeea-961dbb8618cc_characterizeImage_1124710_36.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d1431c49-73d9-41f4-aeea-961dbb8618cc_characterizeImage_1124710_36.stderr parsl.dataflow.dflow INFO: Standard output for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/4513d049-1425-42ff-ad6c-646ea7c1f5ad_isr_1124710_36.stdout parsl.dataflow.dflow INFO: Standard error for task 696 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/4513d049-1425-42ff-ad6c-646ea7c1f5ad_isr_1124710_36.stderr parsl.dataflow.dflow INFO: Task 678 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 427cc7b3a04ec7f88b59d09a4dc7517c with result from task 678 parsl.dataflow.dflow DEBUG: Task 678 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 679 has memoization hash f1ad4969f921e2a06e3bcd8cf2348940 parsl.dataflow.memoization INFO: Task 679 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297398a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 679 try 0 launched on executor multi with executor id 546 parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e413339b-23d9-4807-842b-f988cd70a5c6_characterizeImage_1124710_3.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e413339b-23d9-4807-842b-f988cd70a5c6_characterizeImage_1124710_3.stderr parsl.dataflow.dflow INFO: Standard output for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/0b9172f4-99b7-4c4d-adf0-9d9c26a48318_isr_1124710_3.stdout parsl.dataflow.dflow INFO: Standard error for task 678 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/0b9172f4-99b7-4c4d-adf0-9d9c26a48318_isr_1124710_3.stderr parsl.dataflow.dflow INFO: Task 684 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6f3d87e017ff764cb604570c9e070093 with result from task 684 parsl.dataflow.dflow DEBUG: Task 684 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 685 has memoization hash 4345d15509a76aba6aec2b5e3942c009 parsl.dataflow.memoization INFO: Task 685 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529739da0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 685 try 0 launched on executor multi with executor id 547 parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/089dfc57-4ecd-4cc7-bf9f-c929da6d593a_characterizeImage_1124707_4.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/089dfc57-4ecd-4cc7-bf9f-c929da6d593a_characterizeImage_1124707_4.stderr parsl.dataflow.dflow INFO: Standard output for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/d0c753d0-a635-4579-8f87-866372527993_isr_1124707_4.stdout parsl.dataflow.dflow INFO: Standard error for task 684 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/d0c753d0-a635-4579-8f87-866372527993_isr_1124707_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:21 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 657 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b266794165b0b36d5414d6355b208d39 with result from task 657 parsl.dataflow.dflow DEBUG: Task 657 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 658 has memoization hash f493f5a01883ad22b7605c8cedafd302 parsl.dataflow.memoization INFO: Task 658 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529738400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 658 try 0 launched on executor multi with executor id 548 parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/c8bcd4db-c7b5-498f-ac27-a5b26d09cfb3_characterizeImage_1124704_30.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/c8bcd4db-c7b5-498f-ac27-a5b26d09cfb3_characterizeImage_1124704_30.stderr parsl.dataflow.dflow INFO: Standard output for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a788dc17-c313-4e92-91e7-55aea9bc489b_isr_1124704_30.stdout parsl.dataflow.dflow INFO: Standard error for task 657 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/a788dc17-c313-4e92-91e7-55aea9bc489b_isr_1124704_30.stderr parsl.dataflow.dflow INFO: Task 726 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98afd35ebf84fa98bc63ed3875e3d43f with result from task 726 parsl.dataflow.dflow DEBUG: Task 726 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 727 has memoization hash 3fd48c0d2f4a6ef63d8790447327958f parsl.dataflow.memoization INFO: Task 727 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 727 try 0 launched on executor multi with executor id 549 parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/20715782-354a-4bcc-bd53-0ce506a63acd_characterizeImage_1124828_49.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/20715782-354a-4bcc-bd53-0ce506a63acd_characterizeImage_1124828_49.stderr parsl.dataflow.dflow INFO: Standard output for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/7dbbada0-cbb7-4ad9-ae4a-4c00c394e71f_isr_1124828_49.stdout parsl.dataflow.dflow INFO: Standard error for task 726 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/7dbbada0-cbb7-4ad9-ae4a-4c00c394e71f_isr_1124828_49.stderr parsl.dataflow.dflow INFO: Task 741 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6554cb5d6023580a5b15155e769549af with result from task 741 parsl.dataflow.dflow DEBUG: Task 741 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 742 has memoization hash 0b5eabe388ec5c19f3af78c3cff42b47 parsl.dataflow.memoization INFO: Task 742 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977d620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 742 try 0 launched on executor multi with executor id 550 parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/5bcfedb6-044c-42fe-aa88-9386712df486_characterizeImage_1124704_24.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/5bcfedb6-044c-42fe-aa88-9386712df486_characterizeImage_1124704_24.stderr parsl.dataflow.dflow INFO: Standard output for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/85c80fc7-8818-4145-b999-b08d05a0c3c9_isr_1124704_24.stdout parsl.dataflow.dflow INFO: Standard error for task 741 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/85c80fc7-8818-4145-b999-b08d05a0c3c9_isr_1124704_24.stderr parsl.dataflow.dflow INFO: Task 735 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8be0a07500744a775227e6ad4ee0f01 with result from task 735 parsl.dataflow.dflow DEBUG: Task 735 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 736 has memoization hash 419e98f5e12ac6a05e8a80d35b48c3a1 parsl.dataflow.memoization INFO: Task 736 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977f7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 736 try 0 launched on executor multi with executor id 551 parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/3dd8ac04-7e6e-411d-8018-f69513f330ab_characterizeImage_1124828_38.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/3dd8ac04-7e6e-411d-8018-f69513f330ab_characterizeImage_1124828_38.stderr parsl.dataflow.dflow INFO: Standard output for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8bde51f5-2408-4ab4-a7eb-df59df48d04e_isr_1124828_38.stdout parsl.dataflow.dflow INFO: Standard error for task 735 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/8bde51f5-2408-4ab4-a7eb-df59df48d04e_isr_1124828_38.stderr parsl.dataflow.dflow INFO: Task 666 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d7bcf65bb59e97f016972377a07809a with result from task 666 parsl.dataflow.dflow DEBUG: Task 666 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 667 has memoization hash 7dbbd5a6985a7fdf6b22b121ffc161c5 parsl.dataflow.memoization INFO: Task 667 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 667 try 0 launched on executor multi with executor id 552 parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f98f063c-c4ed-4aaa-8a55-38dfbbc9a293_characterizeImage_1124704_62.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f98f063c-c4ed-4aaa-8a55-38dfbbc9a293_characterizeImage_1124704_62.stderr parsl.dataflow.dflow INFO: Standard output for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/7d51aa65-0980-4a66-90ad-9bf2fb2eca19_isr_1124704_62.stdout parsl.dataflow.dflow INFO: Standard error for task 666 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/7d51aa65-0980-4a66-90ad-9bf2fb2eca19_isr_1124704_62.stderr parsl.dataflow.dflow INFO: Task 783 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 186dfc876febe3212f1be1d80843b5dd with result from task 783 parsl.dataflow.dflow DEBUG: Task 783 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 784 has memoization hash acb35b91f995e2b25bf3316ea832b914 parsl.dataflow.memoization INFO: Task 784 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 784 try 0 launched on executor multi with executor id 553 parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7b26fa5d-acd5-449b-b1a5-1bfe3eed3e1c_characterizeImage_1124710_20.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7b26fa5d-acd5-449b-b1a5-1bfe3eed3e1c_characterizeImage_1124710_20.stderr parsl.dataflow.dflow INFO: Standard output for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f62bc6f4-b871-42e3-96ab-9ca802bb072d_isr_1124710_20.stdout parsl.dataflow.dflow INFO: Standard error for task 783 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f62bc6f4-b871-42e3-96ab-9ca802bb072d_isr_1124710_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:22 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 729 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7261c308acf90daa0af9c70b6d44e63 with result from task 729 parsl.dataflow.dflow DEBUG: Task 729 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 730 has memoization hash 54742a04aaaacdff68ed1fd97dc7c950 parsl.dataflow.memoization INFO: Task 730 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977ff60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 730 try 0 launched on executor multi with executor id 554 parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/2d62d27b-6ea3-4f0f-92d6-c73c201d2d5f_characterizeImage_1124831_62.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/2d62d27b-6ea3-4f0f-92d6-c73c201d2d5f_characterizeImage_1124831_62.stderr parsl.dataflow.dflow INFO: Standard output for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/80f2c890-1be4-429d-a346-780b74e1f902_isr_1124831_62.stdout parsl.dataflow.dflow INFO: Standard error for task 729 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/80f2c890-1be4-429d-a346-780b74e1f902_isr_1124831_62.stderr parsl.dataflow.dflow INFO: Task 648 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9720b36f2ce7c3b316e71ce07098a73e with result from task 648 parsl.dataflow.dflow DEBUG: Task 648 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 649 has memoization hash 65c739d038cf4db0aa2a0d499a6b7d68 parsl.dataflow.memoization INFO: Task 649 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ebc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 649 try 0 launched on executor multi with executor id 555 parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/dd274776-ab8b-4651-acc9-286dae18a29b_characterizeImage_1124831_7.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/dd274776-ab8b-4651-acc9-286dae18a29b_characterizeImage_1124831_7.stderr parsl.dataflow.dflow INFO: Standard output for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/c5e1ed19-060b-4e36-9f2a-a3efd3250b5e_isr_1124831_7.stdout parsl.dataflow.dflow INFO: Standard error for task 648 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/c5e1ed19-060b-4e36-9f2a-a3efd3250b5e_isr_1124831_7.stderr parsl.dataflow.dflow INFO: Task 789 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5eb45fb52da4cfc21327b42b54d2de2 with result from task 789 parsl.dataflow.dflow DEBUG: Task 789 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 790 has memoization hash 2ffeb30de1fc5dc813d52717a0430b1f parsl.dataflow.memoization INFO: Task 790 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977cd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 790 try 0 launched on executor multi with executor id 556 parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b8daa3ec-07ba-44ac-9d00-2e5cbd0f7ac1_characterizeImage_1124704_47.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b8daa3ec-07ba-44ac-9d00-2e5cbd0f7ac1_characterizeImage_1124704_47.stderr parsl.dataflow.dflow INFO: Standard output for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/be0a15b7-d0bd-4356-8a07-6b064a480bac_isr_1124704_47.stdout parsl.dataflow.dflow INFO: Standard error for task 789 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/be0a15b7-d0bd-4356-8a07-6b064a480bac_isr_1124704_47.stderr parsl.dataflow.dflow INFO: Task 777 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1aff978494c26599f1a5441054e6fd3 with result from task 777 parsl.dataflow.dflow DEBUG: Task 777 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 778 has memoization hash 940501a21a0b14b20d64c638e63639c0 parsl.dataflow.memoization INFO: Task 778 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c80e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 778 try 0 launched on executor multi with executor id 557 parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/28d32b11-7189-4984-a945-5ef0291defb4_characterizeImage_1124828_3.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/28d32b11-7189-4984-a945-5ef0291defb4_characterizeImage_1124828_3.stderr parsl.dataflow.dflow INFO: Standard output for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/5f7cc30b-538c-4995-8110-58fdd6eaa9b9_isr_1124828_3.stdout parsl.dataflow.dflow INFO: Standard error for task 777 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/5f7cc30b-538c-4995-8110-58fdd6eaa9b9_isr_1124828_3.stderr parsl.dataflow.dflow INFO: Task 669 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5bf537b5b6d098f6e20a68c4f80be80a with result from task 669 parsl.dataflow.dflow DEBUG: Task 669 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 670 has memoization hash cd9cf50cc26323bd100a8a858396b430 parsl.dataflow.memoization INFO: Task 670 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529738cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 670 try 0 launched on executor multi with executor id 558 parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/c0167ffd-2eb8-4087-a2ce-c415120cc0d1_characterizeImage_1124828_47.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/c0167ffd-2eb8-4087-a2ce-c415120cc0d1_characterizeImage_1124828_47.stderr parsl.dataflow.dflow INFO: Standard output for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/f1a1a4ca-4c3d-4adf-9038-cadcca3e1637_isr_1124828_47.stdout parsl.dataflow.dflow INFO: Standard error for task 669 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/f1a1a4ca-4c3d-4adf-9038-cadcca3e1637_isr_1124828_47.stderr parsl.dataflow.dflow INFO: Task 747 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8bae23e1f842188cfac13dc593bdea7 with result from task 747 parsl.dataflow.dflow DEBUG: Task 747 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 748 has memoization hash 9f59ee1ddb3f26fadbccfa6d29e6533d parsl.dataflow.memoization INFO: Task 748 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977dbc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 748 try 0 launched on executor multi with executor id 559 parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e546f862-4a02-4e0f-b4b3-bed4a3b43471_characterizeImage_1124707_61.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e546f862-4a02-4e0f-b4b3-bed4a3b43471_characterizeImage_1124707_61.stderr parsl.dataflow.dflow INFO: Standard output for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/82c4c721-a39f-453d-8b23-33be8c6efc29_isr_1124707_61.stdout parsl.dataflow.dflow INFO: Standard error for task 747 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/82c4c721-a39f-453d-8b23-33be8c6efc29_isr_1124707_61.stderr parsl.dataflow.dflow INFO: Task 840 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f50f387e0fbdb19a82bdc41eed228f32 with result from task 840 parsl.dataflow.dflow DEBUG: Task 840 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 841 has memoization hash 84050f9a282564ff9a17ec3624b94f74 parsl.dataflow.memoization INFO: Task 841 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296084a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 841 try 0 launched on executor multi with executor id 560 parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/6531013e-4d6c-4bc2-95e1-e61b9e65aaf8_characterizeImage_1124828_39.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/6531013e-4d6c-4bc2-95e1-e61b9e65aaf8_characterizeImage_1124828_39.stderr parsl.dataflow.dflow INFO: Standard output for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/1ac21610-c6b5-4142-9def-ac7ba86c4d15_isr_1124828_39.stdout parsl.dataflow.dflow INFO: Standard error for task 840 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/1ac21610-c6b5-4142-9def-ac7ba86c4d15_isr_1124828_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:25 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 918 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 897dc8e52ad7c37512d7d3d1f42567ed with result from task 918 parsl.dataflow.dflow DEBUG: Task 918 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 919 has memoization hash 2715dc843bb514ca10f76f5df9082302 parsl.dataflow.memoization INFO: Task 919 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529664ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 919 try 0 launched on executor multi with executor id 561 parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/ae27b3bf-54fa-4f3b-a843-839ce9c4fb01_characterizeImage_1124704_48.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/ae27b3bf-54fa-4f3b-a843-839ce9c4fb01_characterizeImage_1124704_48.stderr parsl.dataflow.dflow INFO: Standard output for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/fa85e704-49b8-4af3-b102-b51eb508080c_isr_1124704_48.stdout parsl.dataflow.dflow INFO: Standard error for task 918 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/fa85e704-49b8-4af3-b102-b51eb508080c_isr_1124704_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:26 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 7 try 0 failed parsl.dataflow.dflow INFO: Task 7 marked for retry parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bdc00ba4-6611-4e7e-9998-eb57597fa382_characterizeImage_1124704_2.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bdc00ba4-6611-4e7e-9998-eb57597fa382_characterizeImage_1124704_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 7 has memoization hash 88040c6e7c7cf0d31ccdaf37bb808a6e parsl.dataflow.memoization INFO: Task 7 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad496c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 7 try 1 launched on executor multi with executor id 562 parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bdc00ba4-6611-4e7e-9998-eb57597fa382_characterizeImage_1124704_2.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bdc00ba4-6611-4e7e-9998-eb57597fa382_characterizeImage_1124704_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 16 try 0 failed parsl.dataflow.dflow INFO: Task 16 marked for retry parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/98428587-0f80-48cf-b5ae-ec3269d4888b_characterizeImage_1124828_2.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/98428587-0f80-48cf-b5ae-ec3269d4888b_characterizeImage_1124828_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 16 has memoization hash 33bb2dcd21588a19fda78b0824addbe2 parsl.dataflow.memoization INFO: Task 16 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 16 try 1 launched on executor multi with executor id 563 parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/98428587-0f80-48cf-b5ae-ec3269d4888b_characterizeImage_1124828_2.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/98428587-0f80-48cf-b5ae-ec3269d4888b_characterizeImage_1124828_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:27 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:28 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 762 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65b97aca65951d91408d41b5b4a97668 with result from task 762 parsl.dataflow.dflow DEBUG: Task 762 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 763 has memoization hash 7165f85bcc8b47683e89b02d92a95fa7 parsl.dataflow.memoization INFO: Task 763 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 763 try 0 launched on executor multi with executor id 564 parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/62bb184b-a588-43f6-a9d9-bc63c94c64a2_characterizeImage_1124828_40.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/62bb184b-a588-43f6-a9d9-bc63c94c64a2_characterizeImage_1124828_40.stderr parsl.dataflow.dflow INFO: Standard output for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/5e753b90-61a3-4aa5-a648-f2a32e4d2925_isr_1124828_40.stdout parsl.dataflow.dflow INFO: Standard error for task 762 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/5e753b90-61a3-4aa5-a648-f2a32e4d2925_isr_1124828_40.stderr parsl.dataflow.dflow INFO: Task 774 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a39061f13ef9f9f1416b6e4b9453a497 with result from task 774 parsl.dataflow.dflow DEBUG: Task 774 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 775 has memoization hash d809ffc247d8b1271e1f6a7c3b65b02e parsl.dataflow.memoization INFO: Task 775 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 775 try 0 launched on executor multi with executor id 565 parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/c72d92d3-6863-4b77-87c0-c01d113e0b1c_characterizeImage_1124710_8.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/c72d92d3-6863-4b77-87c0-c01d113e0b1c_characterizeImage_1124710_8.stderr parsl.dataflow.dflow INFO: Standard output for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/0cad93c9-ab9f-46de-a094-cad87202ff82_isr_1124710_8.stdout parsl.dataflow.dflow INFO: Standard error for task 774 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/0cad93c9-ab9f-46de-a094-cad87202ff82_isr_1124710_8.stderr parsl.dataflow.dflow INFO: Task 771 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 547e279e91c7f6b6d4ed65d7a662ffa6 with result from task 771 parsl.dataflow.dflow DEBUG: Task 771 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash 3de12400f6b8a0bb3edf4a850a7177da parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 0 launched on executor multi with executor id 566 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/142f6ce8-f81b-476d-ac52-5acf2bf1c6d5_characterizeImage_1124831_2.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/142f6ce8-f81b-476d-ac52-5acf2bf1c6d5_characterizeImage_1124831_2.stderr parsl.dataflow.dflow INFO: Standard output for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/12179eb3-1fb3-4168-80c0-4a46c1d1932c_isr_1124831_2.stdout parsl.dataflow.dflow INFO: Standard error for task 771 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/12179eb3-1fb3-4168-80c0-4a46c1d1932c_isr_1124831_2.stderr parsl.dataflow.dflow INFO: Task 759 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 70dfa67ef58591cd775c9e0ab5175bd4 with result from task 759 parsl.dataflow.dflow DEBUG: Task 759 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 760 has memoization hash edcf7ef07e7131fc231c5082c3281f2d parsl.dataflow.memoization INFO: Task 760 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977e8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 760 try 0 launched on executor multi with executor id 567 parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/ad3dd203-1a65-4a9c-ad4b-016f013d556b_characterizeImage_1124704_51.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/ad3dd203-1a65-4a9c-ad4b-016f013d556b_characterizeImage_1124704_51.stderr parsl.dataflow.dflow INFO: Standard output for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/076460fd-ba33-42ad-a636-79f62af670f3_isr_1124704_51.stdout parsl.dataflow.dflow INFO: Standard error for task 759 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/076460fd-ba33-42ad-a636-79f62af670f3_isr_1124704_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:30 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 837 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c782002e54dcd8c35fb740f7e4da654 with result from task 837 parsl.dataflow.dflow DEBUG: Task 837 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 838 has memoization hash 9acecf7e216a8b742aecf318d672ce76 parsl.dataflow.memoization INFO: Task 838 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cb600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 838 try 0 launched on executor multi with executor id 568 parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/4dfab188-f9c9-4443-a148-77afab903adc_characterizeImage_1124710_14.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/4dfab188-f9c9-4443-a148-77afab903adc_characterizeImage_1124710_14.stderr parsl.dataflow.dflow INFO: Standard output for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/3ed84469-5fcd-4e63-8d32-84940fa60bf5_isr_1124710_14.stdout parsl.dataflow.dflow INFO: Standard error for task 837 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/3ed84469-5fcd-4e63-8d32-84940fa60bf5_isr_1124710_14.stderr parsl.dataflow.dflow INFO: Task 831 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ff79567559828ebffb86c750a9d45cbb with result from task 831 parsl.dataflow.dflow DEBUG: Task 831 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 832 has memoization hash 4e55c91c43ca41ddcc1b3d93dc20bae2 parsl.dataflow.memoization INFO: Task 832 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 832 try 0 launched on executor multi with executor id 569 parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/4351cc4b-11d2-458b-a7c8-e6484c15b62d_characterizeImage_1124704_13.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/4351cc4b-11d2-458b-a7c8-e6484c15b62d_characterizeImage_1124704_13.stderr parsl.dataflow.dflow INFO: Standard output for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/8a74add7-1c93-4b1f-88b7-1bb992c8c5fe_isr_1124704_13.stdout parsl.dataflow.dflow INFO: Standard error for task 831 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/8a74add7-1c93-4b1f-88b7-1bb992c8c5fe_isr_1124704_13.stderr parsl.dataflow.dflow INFO: Task 849 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c94c83602169a1ad284392725f3f2449 with result from task 849 parsl.dataflow.dflow DEBUG: Task 849 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 850 has memoization hash 183ac088bf5d4c1100eee0361fea52d8 parsl.dataflow.memoization INFO: Task 850 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 850 try 0 launched on executor multi with executor id 570 parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/766b7761-173f-4ef2-982b-b439c89e3180_characterizeImage_1124828_8.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/766b7761-173f-4ef2-982b-b439c89e3180_characterizeImage_1124828_8.stderr parsl.dataflow.dflow INFO: Standard output for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/2cb0b756-b53b-4381-b683-45f26c556322_isr_1124828_8.stdout parsl.dataflow.dflow INFO: Standard error for task 849 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/2cb0b756-b53b-4381-b683-45f26c556322_isr_1124828_8.stderr parsl.dataflow.dflow INFO: Task 855 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee9c0f38279566567f0a824fcfa96721 with result from task 855 parsl.dataflow.dflow DEBUG: Task 855 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 856 has memoization hash 0fde0a0951042f1843737dc045f96e3e parsl.dataflow.memoization INFO: Task 856 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296094e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 856 try 0 launched on executor multi with executor id 571 parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/23b2cc8b-c6cb-409e-8ad1-a85945b1e7dd_characterizeImage_1124710_5.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/23b2cc8b-c6cb-409e-8ad1-a85945b1e7dd_characterizeImage_1124710_5.stderr parsl.dataflow.dflow INFO: Standard output for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f26304c3-ba6a-49d3-bde4-01f1b6ffb869_isr_1124710_5.stdout parsl.dataflow.dflow INFO: Standard error for task 855 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/f26304c3-ba6a-49d3-bde4-01f1b6ffb869_isr_1124710_5.stderr parsl.dataflow.dflow INFO: Task 834 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d38121917c4e7dbca667cd60fdb0a2a8 with result from task 834 parsl.dataflow.dflow DEBUG: Task 834 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 835 has memoization hash 056474afebd676d12e87e5cb641cdd9e parsl.dataflow.memoization INFO: Task 835 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cb560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 835 try 0 launched on executor multi with executor id 572 parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/647a9612-c987-48ed-9ae6-e5597189058d_characterizeImage_1124707_57.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/647a9612-c987-48ed-9ae6-e5597189058d_characterizeImage_1124707_57.stderr parsl.dataflow.dflow INFO: Standard output for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/1541a5d5-0501-4ca6-9db1-5abe744bd9e6_isr_1124707_57.stdout parsl.dataflow.dflow INFO: Standard error for task 834 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/1541a5d5-0501-4ca6-9db1-5abe744bd9e6_isr_1124707_57.stderr parsl.dataflow.dflow INFO: Task 828 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 935335151d379fa62e99047fd22c93ef with result from task 828 parsl.dataflow.dflow DEBUG: Task 828 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 829 has memoization hash be50399ae7b14e6bbbf16a73d1c64f2d parsl.dataflow.memoization INFO: Task 829 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 829 try 0 launched on executor multi with executor id 573 parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/634ccfbd-7010-464e-828f-88a4a78c5556_characterizeImage_1124828_56.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/634ccfbd-7010-464e-828f-88a4a78c5556_characterizeImage_1124828_56.stderr parsl.dataflow.dflow INFO: Standard output for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/0c727113-2835-4683-b1d8-55bd0c235af0_isr_1124828_56.stdout parsl.dataflow.dflow INFO: Standard error for task 828 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/0c727113-2835-4683-b1d8-55bd0c235af0_isr_1124828_56.stderr parsl.dataflow.dflow INFO: Task 861 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4bd283226ef672be18fa7d86bd80241 with result from task 861 parsl.dataflow.dflow DEBUG: Task 861 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 862 has memoization hash ee6d44b90bd39946586a0b4262e2b2ea parsl.dataflow.memoization INFO: Task 862 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529609b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 862 try 0 launched on executor multi with executor id 574 parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/a9dc3d83-6d39-43bd-b9cd-1c69200cefbe_characterizeImage_1124707_28.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/a9dc3d83-6d39-43bd-b9cd-1c69200cefbe_characterizeImage_1124707_28.stderr parsl.dataflow.dflow INFO: Standard output for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/8fe87486-8b36-4971-aa3f-3cff0082839f_isr_1124707_28.stdout parsl.dataflow.dflow INFO: Standard error for task 861 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/8fe87486-8b36-4971-aa3f-3cff0082839f_isr_1124707_28.stderr parsl.dataflow.dflow INFO: Task 819 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e1e60295c4c54b05dc79a6eb638729a7 with result from task 819 parsl.dataflow.dflow DEBUG: Task 819 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 820 has memoization hash 2dc3625834640182e3c2fc4ae98b5780 parsl.dataflow.memoization INFO: Task 820 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295ca5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 820 try 0 launched on executor multi with executor id 575 parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/24ffecc0-0d98-4a7b-b390-dfbf20733bd5_characterizeImage_1124707_52.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/24ffecc0-0d98-4a7b-b390-dfbf20733bd5_characterizeImage_1124707_52.stderr parsl.dataflow.dflow INFO: Standard output for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/00c6d61c-c4e2-48d4-b536-99ffeb7cb84c_isr_1124707_52.stdout parsl.dataflow.dflow INFO: Standard error for task 819 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/00c6d61c-c4e2-48d4-b536-99ffeb7cb84c_isr_1124707_52.stderr parsl.dataflow.dflow INFO: Task 885 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54229a8d2f3535d951e037d6e371f12c with result from task 885 parsl.dataflow.dflow DEBUG: Task 885 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 886 has memoization hash 89973e3adcc9f336fdcece7e8f929c91 parsl.dataflow.memoization INFO: Task 886 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 886 try 0 launched on executor multi with executor id 576 parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b02cf82c-6973-4a16-8734-78394f197f9c_characterizeImage_1124704_57.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b02cf82c-6973-4a16-8734-78394f197f9c_characterizeImage_1124704_57.stderr parsl.dataflow.dflow INFO: Standard output for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/14b5c581-1ad6-4c8c-9606-6ff755abe346_isr_1124704_57.stdout parsl.dataflow.dflow INFO: Standard error for task 885 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/14b5c581-1ad6-4c8c-9606-6ff755abe346_isr_1124704_57.stderr parsl.dataflow.dflow INFO: Task 756 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c2437b7c55a5e7790b8cefabc086e3f with result from task 756 parsl.dataflow.dflow DEBUG: Task 756 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 757 has memoization hash 7529792690d62d7f4b00e51fefca08b4 parsl.dataflow.memoization INFO: Task 757 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977e5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 757 try 0 launched on executor multi with executor id 577 parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/38d8d024-8186-409f-8a89-25bd4a9d97de_characterizeImage_1124707_7.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/38d8d024-8186-409f-8a89-25bd4a9d97de_characterizeImage_1124707_7.stderr parsl.dataflow.dflow INFO: Standard output for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/96bc151b-db8b-4c47-bfbc-5ef65354fbfd_isr_1124707_7.stdout parsl.dataflow.dflow INFO: Standard error for task 756 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/96bc151b-db8b-4c47-bfbc-5ef65354fbfd_isr_1124707_7.stderr parsl.dataflow.dflow INFO: Task 765 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 966c6b6c7d6642062921e2fe3f570ec6 with result from task 765 parsl.dataflow.dflow DEBUG: Task 765 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 766 has memoization hash 321a37dc4ea997c54169ac4568ee2a17 parsl.dataflow.memoization INFO: Task 766 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 766 try 0 launched on executor multi with executor id 578 parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/1da3ea7d-9792-4592-9dcf-2b2810d8639d_characterizeImage_1124704_31.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/1da3ea7d-9792-4592-9dcf-2b2810d8639d_characterizeImage_1124704_31.stderr parsl.dataflow.dflow INFO: Standard output for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/0f0a93d9-5b14-4ef1-bbec-dc820aa026dd_isr_1124704_31.stdout parsl.dataflow.dflow INFO: Standard error for task 765 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/0f0a93d9-5b14-4ef1-bbec-dc820aa026dd_isr_1124704_31.stderr parsl.dataflow.dflow INFO: Task 807 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce3a6eeb951cbe33d44df365eb6a1e5d with result from task 807 parsl.dataflow.dflow DEBUG: Task 807 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 808 has memoization hash 062f990c1a55d1953966a4614d5661a8 parsl.dataflow.memoization INFO: Task 808 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cbb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 808 try 0 launched on executor multi with executor id 579 parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/44c96ce0-af4b-457d-a9d1-7e95db4baa64_characterizeImage_1124828_57.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/44c96ce0-af4b-457d-a9d1-7e95db4baa64_characterizeImage_1124828_57.stderr parsl.dataflow.dflow INFO: Standard output for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/32b5c052-efbf-435e-90d6-c1ed903f73d5_isr_1124828_57.stdout parsl.dataflow.dflow INFO: Standard error for task 807 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/32b5c052-efbf-435e-90d6-c1ed903f73d5_isr_1124828_57.stderr parsl.dataflow.dflow INFO: Task 795 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5862c5eb1df8306a6252273f81f21499 with result from task 795 parsl.dataflow.dflow DEBUG: Task 795 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 796 has memoization hash 807d63d0413ae0202eb0b3ad41ae0f70 parsl.dataflow.memoization INFO: Task 796 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c9440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 796 try 0 launched on executor multi with executor id 580 parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f29dedc8-699e-4eca-a2e2-95164dcbe728_characterizeImage_1124710_13.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f29dedc8-699e-4eca-a2e2-95164dcbe728_characterizeImage_1124710_13.stderr parsl.dataflow.dflow INFO: Standard output for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/e4ba0759-e842-43ed-8b4e-8c5d0417871c_isr_1124710_13.stdout parsl.dataflow.dflow INFO: Standard error for task 795 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/e4ba0759-e842-43ed-8b4e-8c5d0417871c_isr_1124710_13.stderr parsl.dataflow.dflow INFO: Task 792 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 97015cfbe4a098943862e7b19f7039ac with result from task 792 parsl.dataflow.dflow DEBUG: Task 792 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 793 has memoization hash 7403a737f5906e6b7466de36ffac60ac parsl.dataflow.memoization INFO: Task 793 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c9120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 793 try 0 launched on executor multi with executor id 581 parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/cb0357eb-001e-45a8-933b-d478e4a171be_characterizeImage_1124707_45.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/cb0357eb-001e-45a8-933b-d478e4a171be_characterizeImage_1124707_45.stderr parsl.dataflow.dflow INFO: Standard output for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/3f5d340a-a590-4ea2-82de-da5d026fe97a_isr_1124707_45.stdout parsl.dataflow.dflow INFO: Standard error for task 792 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/3f5d340a-a590-4ea2-82de-da5d026fe97a_isr_1124707_45.stderr parsl.dataflow.dflow INFO: Task 825 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d653ccb16b1760810804ee4e36614527 with result from task 825 parsl.dataflow.dflow DEBUG: Task 825 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 826 has memoization hash f8e345a93057fcae6008c8e2461fa8c0 parsl.dataflow.memoization INFO: Task 826 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295caac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 826 try 0 launched on executor multi with executor id 582 parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f92cb868-7e4b-4405-946b-48e6d8c5edf5_characterizeImage_1124704_32.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f92cb868-7e4b-4405-946b-48e6d8c5edf5_characterizeImage_1124704_32.stderr parsl.dataflow.dflow INFO: Standard output for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/5832ed04-5208-4c94-a871-866fbf40aa48_isr_1124704_32.stdout parsl.dataflow.dflow INFO: Standard error for task 825 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/5832ed04-5208-4c94-a871-866fbf40aa48_isr_1124704_32.stderr parsl.dataflow.dflow INFO: Task 753 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d43ff39b34515394b8a4880338072d3 with result from task 753 parsl.dataflow.dflow DEBUG: Task 753 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 754 has memoization hash 205400159d6ab994bee336b47a6fa6bb parsl.dataflow.memoization INFO: Task 754 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 754 try 0 launched on executor multi with executor id 583 parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b92cb915-c845-4826-a96c-c50fab7d46a3_characterizeImage_1124704_38.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b92cb915-c845-4826-a96c-c50fab7d46a3_characterizeImage_1124704_38.stderr parsl.dataflow.dflow INFO: Standard output for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/0d5d7c77-f177-4adc-bc18-f2e4bb1cf767_isr_1124704_38.stdout parsl.dataflow.dflow INFO: Standard error for task 753 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/0d5d7c77-f177-4adc-bc18-f2e4bb1cf767_isr_1124704_38.stderr parsl.dataflow.dflow INFO: Task 927 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38870b2a1769fcabd7c5f5b4b0caeacf with result from task 927 parsl.dataflow.dflow DEBUG: Task 927 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 928 has memoization hash 011d3b2f31cc4ed785df6813668fcfc0 parsl.dataflow.memoization INFO: Task 928 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296658a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 928 try 0 launched on executor multi with executor id 584 parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d23f19a8-2a87-444e-94f1-42ef0284233a_characterizeImage_1124704_36.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d23f19a8-2a87-444e-94f1-42ef0284233a_characterizeImage_1124704_36.stderr parsl.dataflow.dflow INFO: Standard output for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/8e66cb82-8ec5-4efe-b084-f47e7b675fdc_isr_1124704_36.stdout parsl.dataflow.dflow INFO: Standard error for task 927 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/8e66cb82-8ec5-4efe-b084-f47e7b675fdc_isr_1124704_36.stderr parsl.dataflow.dflow INFO: Task 813 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d2fe7ba6d429648c0be2ab6af685b75 with result from task 813 parsl.dataflow.dflow DEBUG: Task 813 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 814 has memoization hash ab981d1f8a79c3fd8430c24f4f2939a7 parsl.dataflow.memoization INFO: Task 814 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 814 try 0 launched on executor multi with executor id 585 parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b5f5bbb4-aef5-4223-a9f3-2a6fd12045b8_characterizeImage_1124710_38.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b5f5bbb4-aef5-4223-a9f3-2a6fd12045b8_characterizeImage_1124710_38.stderr parsl.dataflow.dflow INFO: Standard output for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/49c4cc2d-d1a7-48a5-b87d-24198533e99c_isr_1124710_38.stdout parsl.dataflow.dflow INFO: Standard error for task 813 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/49c4cc2d-d1a7-48a5-b87d-24198533e99c_isr_1124710_38.stderr parsl.dataflow.dflow INFO: Task 801 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d99defbc97ae936b6322fa878b667aec with result from task 801 parsl.dataflow.dflow DEBUG: Task 801 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 802 has memoization hash 374c1b87d7cae629d8ffafbcd1921b60 parsl.dataflow.memoization INFO: Task 802 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cbba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 802 try 0 launched on executor multi with executor id 586 parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/573f5f93-aa7f-4135-b6be-7847371f0a26_characterizeImage_1124831_41.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/573f5f93-aa7f-4135-b6be-7847371f0a26_characterizeImage_1124831_41.stderr parsl.dataflow.dflow INFO: Standard output for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/c84f2cd4-6afd-4488-80f8-da21ebdffa4e_isr_1124831_41.stdout parsl.dataflow.dflow INFO: Standard error for task 801 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/c84f2cd4-6afd-4488-80f8-da21ebdffa4e_isr_1124831_41.stderr parsl.dataflow.dflow INFO: Task 864 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41cff2039bedb47c2ba5c34ca40c99bd with result from task 864 parsl.dataflow.dflow DEBUG: Task 864 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 865 has memoization hash 47791c3f3014497d052a73ad5f62820f parsl.dataflow.memoization INFO: Task 865 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529609e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 865 try 0 launched on executor multi with executor id 587 parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/83d6e29e-a037-4581-a909-06a98e7099cb_characterizeImage_1124707_14.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/83d6e29e-a037-4581-a909-06a98e7099cb_characterizeImage_1124707_14.stderr parsl.dataflow.dflow INFO: Standard output for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/80ea5f0c-0c6d-4101-acf9-d5e6cd71af36_isr_1124707_14.stdout parsl.dataflow.dflow INFO: Standard error for task 864 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/80ea5f0c-0c6d-4101-acf9-d5e6cd71af36_isr_1124707_14.stderr parsl.dataflow.dflow INFO: Task 804 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e9199a80489cb5d4375f9488a32978f with result from task 804 parsl.dataflow.dflow DEBUG: Task 804 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 805 has memoization hash 5d1ad48b119534cc3714d2f170368a52 parsl.dataflow.memoization INFO: Task 805 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cb880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 805 try 0 launched on executor multi with executor id 588 parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/76153e92-63c5-4522-81d7-3f9496d766fe_characterizeImage_1124828_4.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/76153e92-63c5-4522-81d7-3f9496d766fe_characterizeImage_1124828_4.stderr parsl.dataflow.dflow INFO: Standard output for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/77bba865-fe20-4808-84aa-7fb6fd672671_isr_1124828_4.stdout parsl.dataflow.dflow INFO: Standard error for task 804 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/77bba865-fe20-4808-84aa-7fb6fd672671_isr_1124828_4.stderr parsl.dataflow.dflow INFO: Task 810 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5d76d8d08a8dd476b69232aabbd50a1 with result from task 810 parsl.dataflow.dflow DEBUG: Task 810 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 811 has memoization hash 674d444084cf2deafe08759f85420d30 parsl.dataflow.memoization INFO: Task 811 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 811 try 0 launched on executor multi with executor id 589 parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/d3a6abeb-3c46-495b-9537-a22f431bed05_characterizeImage_1124828_6.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/d3a6abeb-3c46-495b-9537-a22f431bed05_characterizeImage_1124828_6.stderr parsl.dataflow.dflow INFO: Standard output for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bb53a21b-8b5f-432b-b7da-178d3e3a7d4d_isr_1124828_6.stdout parsl.dataflow.dflow INFO: Standard error for task 810 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/bb53a21b-8b5f-432b-b7da-178d3e3a7d4d_isr_1124828_6.stderr parsl.dataflow.dflow INFO: Task 852 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 720b58d6abfeb578e0b1318a67f9c29c with result from task 852 parsl.dataflow.dflow DEBUG: Task 852 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 853 has memoization hash f6f2f9406a3a9ea8e88263cbb7e597f6 parsl.dataflow.memoization INFO: Task 853 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296091c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 853 try 0 launched on executor multi with executor id 590 parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/2378906a-8096-4c74-b050-bebab7e32796_characterizeImage_1124828_44.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/2378906a-8096-4c74-b050-bebab7e32796_characterizeImage_1124828_44.stderr parsl.dataflow.dflow INFO: Standard output for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/c6722b37-a97c-45f8-986c-2405f8777f44_isr_1124828_44.stdout parsl.dataflow.dflow INFO: Standard error for task 852 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/c6722b37-a97c-45f8-986c-2405f8777f44_isr_1124828_44.stderr parsl.dataflow.dflow INFO: Task 798 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b36b2a7ce497f8cb8ae74acbabb1699 with result from task 798 parsl.dataflow.dflow DEBUG: Task 798 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 799 has memoization hash 33cf5a493c7bf01b294fe90a72f2271d parsl.dataflow.memoization INFO: Task 799 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c9760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 799 try 0 launched on executor multi with executor id 591 parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/03720eed-cdeb-4809-8082-30aa1fdaac84_characterizeImage_1124710_7.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/03720eed-cdeb-4809-8082-30aa1fdaac84_characterizeImage_1124710_7.stderr parsl.dataflow.dflow INFO: Standard output for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a73d3153-97ce-4edc-9720-8ab4ab9b77b4_isr_1124710_7.stdout parsl.dataflow.dflow INFO: Standard error for task 798 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a73d3153-97ce-4edc-9720-8ab4ab9b77b4_isr_1124710_7.stderr parsl.dataflow.dflow INFO: Task 822 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab08e854f1a16c9d0c05c5ff80132a6e with result from task 822 parsl.dataflow.dflow DEBUG: Task 822 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 823 has memoization hash 594233d685b644b2d5e2e53239ae9e5b parsl.dataflow.memoization INFO: Task 823 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295ca7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 823 try 0 launched on executor multi with executor id 592 parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/4f46a892-43a9-4ee0-86b3-7fe4cdbe25cc_characterizeImage_1124831_12.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/4f46a892-43a9-4ee0-86b3-7fe4cdbe25cc_characterizeImage_1124831_12.stderr parsl.dataflow.dflow INFO: Standard output for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/deed4ecc-346b-44ea-8b02-774f822aa767_isr_1124831_12.stdout parsl.dataflow.dflow INFO: Standard error for task 822 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/deed4ecc-346b-44ea-8b02-774f822aa767_isr_1124831_12.stderr parsl.dataflow.dflow INFO: Task 816 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9d2fca43e569bc960bc5ce6f673ae6c5 with result from task 816 parsl.dataflow.dflow DEBUG: Task 816 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 817 has memoization hash 854c64df2d6cba2203c9fb93069fd54a parsl.dataflow.memoization INFO: Task 817 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295ca2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 817 try 0 launched on executor multi with executor id 593 parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/84cf8a55-adc4-4c1b-8289-3a2c77c4d254_characterizeImage_1124831_17.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/84cf8a55-adc4-4c1b-8289-3a2c77c4d254_characterizeImage_1124831_17.stderr parsl.dataflow.dflow INFO: Standard output for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f800fbbf-7199-4a7d-9098-ab1f809652c2_isr_1124831_17.stdout parsl.dataflow.dflow INFO: Standard error for task 816 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f800fbbf-7199-4a7d-9098-ab1f809652c2_isr_1124831_17.stderr parsl.dataflow.dflow INFO: Task 870 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fa7efe08c013be1f6d956c339913660 with result from task 870 parsl.dataflow.dflow DEBUG: Task 870 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 871 has memoization hash b58f18a261b2b86debd8a7d28f6ece21 parsl.dataflow.memoization INFO: Task 871 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960a480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 871 try 0 launched on executor multi with executor id 594 parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/0d7ee9c1-cf96-4649-8fb0-0f16ed7964cf_characterizeImage_1124704_15.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/0d7ee9c1-cf96-4649-8fb0-0f16ed7964cf_characterizeImage_1124704_15.stderr parsl.dataflow.dflow INFO: Standard output for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/5a7ab523-21ef-4720-a8e5-f595b8926e65_isr_1124704_15.stdout parsl.dataflow.dflow INFO: Standard error for task 870 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/5a7ab523-21ef-4720-a8e5-f595b8926e65_isr_1124704_15.stderr parsl.dataflow.dflow INFO: Task 924 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aedc28371ec063e84e7b157e2005efb with result from task 924 parsl.dataflow.dflow DEBUG: Task 924 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 925 has memoization hash 828b317b5ed21103f60753422048e912 parsl.dataflow.memoization INFO: Task 925 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529665580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 925 try 0 launched on executor multi with executor id 595 parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/f706409f-d380-4283-a729-1d6e68dd6bfe_characterizeImage_1124828_7.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/f706409f-d380-4283-a729-1d6e68dd6bfe_characterizeImage_1124828_7.stderr parsl.dataflow.dflow INFO: Standard output for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/3a115b33-73b6-44a2-b974-1baf1f89b2cb_isr_1124828_7.stdout parsl.dataflow.dflow INFO: Standard error for task 924 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/3a115b33-73b6-44a2-b974-1baf1f89b2cb_isr_1124828_7.stderr parsl.dataflow.dflow INFO: Task 846 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d5bfab313b0229dd47f31054995b3d3 with result from task 846 parsl.dataflow.dflow DEBUG: Task 846 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 847 has memoization hash 9c1a5720583214712e584a5241810446 parsl.dataflow.memoization INFO: Task 847 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 847 try 0 launched on executor multi with executor id 596 parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/c1ecdacf-edcb-4edb-825a-c0455d829e44_characterizeImage_1124707_59.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/c1ecdacf-edcb-4edb-825a-c0455d829e44_characterizeImage_1124707_59.stderr parsl.dataflow.dflow INFO: Standard output for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0e6c25f1-00d9-42b7-9c97-de5fd602e977_isr_1124707_59.stdout parsl.dataflow.dflow INFO: Standard error for task 846 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/0e6c25f1-00d9-42b7-9c97-de5fd602e977_isr_1124707_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:32 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 768 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21d2f6fd4b205ffe4fefc902d19541ef with result from task 768 parsl.dataflow.dflow DEBUG: Task 768 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 769 has memoization hash 3fd6a56d3cb50e6534e9c8a515ec0881 parsl.dataflow.memoization INFO: Task 769 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977f4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 769 try 0 launched on executor multi with executor id 597 parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/5627f992-3f4f-419b-97cd-88d9cd8ef0f6_characterizeImage_1124704_59.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/5627f992-3f4f-419b-97cd-88d9cd8ef0f6_characterizeImage_1124704_59.stderr parsl.dataflow.dflow INFO: Standard output for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/c1e93f2a-b4f9-443c-b8bd-802a7f0cfc5c_isr_1124704_59.stdout parsl.dataflow.dflow INFO: Standard error for task 768 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/c1e93f2a-b4f9-443c-b8bd-802a7f0cfc5c_isr_1124704_59.stderr parsl.dataflow.dflow INFO: Task 873 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6769418f65373ab1b5f88e4a0f065d38 with result from task 873 parsl.dataflow.dflow DEBUG: Task 873 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 874 has memoization hash 3d60712c702bf109c1ab970fb060c687 parsl.dataflow.memoization INFO: Task 874 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960a7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 874 try 0 launched on executor multi with executor id 598 parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/72d6d45d-3aa4-47d2-9ffd-af56e4ed7198_characterizeImage_1124831_38.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/72d6d45d-3aa4-47d2-9ffd-af56e4ed7198_characterizeImage_1124831_38.stderr parsl.dataflow.dflow INFO: Standard output for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f1dc2506-5c18-4a32-957b-a4c20890cdb8_isr_1124831_38.stdout parsl.dataflow.dflow INFO: Standard error for task 873 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/f1dc2506-5c18-4a32-957b-a4c20890cdb8_isr_1124831_38.stderr parsl.dataflow.dflow INFO: Task 867 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ffffeff77e868cb39c08f3e0a8b68d1 with result from task 867 parsl.dataflow.dflow DEBUG: Task 867 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 868 has memoization hash 01fe69571c20250909e31a1d68101b01 parsl.dataflow.memoization INFO: Task 868 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960a160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 868 try 0 launched on executor multi with executor id 599 parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/97ede7ed-10e2-4315-94b3-023473b5bdae_characterizeImage_1124831_26.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/97ede7ed-10e2-4315-94b3-023473b5bdae_characterizeImage_1124831_26.stderr parsl.dataflow.dflow INFO: Standard output for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/66c7eb2d-52f5-4164-8e36-58e4e7fa342b_isr_1124831_26.stdout parsl.dataflow.dflow INFO: Standard error for task 867 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/66c7eb2d-52f5-4164-8e36-58e4e7fa342b_isr_1124831_26.stderr parsl.dataflow.dflow INFO: Task 888 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 075de847a088d1544908ba1a16b05fae with result from task 888 parsl.dataflow.dflow DEBUG: Task 888 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 889 has memoization hash ea428097881235b9fb58b0460a44bde8 parsl.dataflow.memoization INFO: Task 889 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 889 try 0 launched on executor multi with executor id 600 parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8413d4a2-8de0-488f-bf68-7a748d99a6fe_characterizeImage_1124707_29.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8413d4a2-8de0-488f-bf68-7a748d99a6fe_characterizeImage_1124707_29.stderr parsl.dataflow.dflow INFO: Standard output for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/172428d8-8195-41f4-8c07-9d0abe0f180a_isr_1124707_29.stdout parsl.dataflow.dflow INFO: Standard error for task 888 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/172428d8-8195-41f4-8c07-9d0abe0f180a_isr_1124707_29.stderr parsl.dataflow.dflow INFO: Task 891 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1368a621a13bd3cdf4bf368d6c4e40c7 with result from task 891 parsl.dataflow.dflow DEBUG: Task 891 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 892 has memoization hash 42dbd900ff8d2c9aa6333e6338bdc0db parsl.dataflow.memoization INFO: Task 892 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960ba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 892 try 0 launched on executor multi with executor id 601 parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/da5c7edd-e3a2-4a1f-bf9e-246c03002aa0_characterizeImage_1124707_23.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/da5c7edd-e3a2-4a1f-bf9e-246c03002aa0_characterizeImage_1124707_23.stderr parsl.dataflow.dflow INFO: Standard output for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/21aa76a4-fc66-435a-bb0a-2f29fb959cd8_isr_1124707_23.stdout parsl.dataflow.dflow INFO: Standard error for task 891 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/21aa76a4-fc66-435a-bb0a-2f29fb959cd8_isr_1124707_23.stderr parsl.dataflow.dflow INFO: Task 882 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e48869f8111427c6044fdff576f709c0 with result from task 882 parsl.dataflow.dflow DEBUG: Task 882 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 883 has memoization hash 0b595716e5928ed9893c40b90aa63140 parsl.dataflow.memoization INFO: Task 883 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 883 try 0 launched on executor multi with executor id 602 parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/fca9eacd-7210-45c5-8537-632f5f202710_characterizeImage_1124828_22.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/fca9eacd-7210-45c5-8537-632f5f202710_characterizeImage_1124828_22.stderr parsl.dataflow.dflow INFO: Standard output for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/fe76f12c-f0e7-4a74-bca4-80910896cbaf_isr_1124828_22.stdout parsl.dataflow.dflow INFO: Standard error for task 882 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/fe76f12c-f0e7-4a74-bca4-80910896cbaf_isr_1124828_22.stderr parsl.dataflow.dflow INFO: Task 921 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c27fd324fcb5da0b939d503f1eb48a4a with result from task 921 parsl.dataflow.dflow DEBUG: Task 921 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 922 has memoization hash 6d8b8c71f3489fe39de044a9c44e5974 parsl.dataflow.memoization INFO: Task 922 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296651c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 922 try 0 launched on executor multi with executor id 603 parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/2187b440-49a9-446e-9ff8-ea79727cec2b_characterizeImage_1124831_48.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/2187b440-49a9-446e-9ff8-ea79727cec2b_characterizeImage_1124831_48.stderr parsl.dataflow.dflow INFO: Standard output for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7ea8b424-1e67-4a98-819a-567ea491fb08_isr_1124831_48.stdout parsl.dataflow.dflow INFO: Standard error for task 921 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/7ea8b424-1e67-4a98-819a-567ea491fb08_isr_1124831_48.stderr parsl.dataflow.dflow INFO: Task 858 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1f1906ef35ef368f33d70cd7d715ef4a with result from task 858 parsl.dataflow.dflow DEBUG: Task 858 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 859 has memoization hash 4e986b3f23f00955fa3496ff503253da parsl.dataflow.memoization INFO: Task 859 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529609800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 859 try 0 launched on executor multi with executor id 604 parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/767d8902-f0c6-46d3-9e1e-cd8974a46f4b_characterizeImage_1124828_52.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/767d8902-f0c6-46d3-9e1e-cd8974a46f4b_characterizeImage_1124828_52.stderr parsl.dataflow.dflow INFO: Standard output for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/2b086f0a-3d53-4043-a10b-a60ecac1ed26_isr_1124828_52.stdout parsl.dataflow.dflow INFO: Standard error for task 858 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/2b086f0a-3d53-4043-a10b-a60ecac1ed26_isr_1124828_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:33 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 876 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c39bddcfeee0996b2338b1ed8a5c2e6 with result from task 876 parsl.dataflow.dflow DEBUG: Task 876 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 877 has memoization hash 54869dc7fe91dbe123fde3f5c16f7d7d parsl.dataflow.memoization INFO: Task 877 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960aac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 877 try 0 launched on executor multi with executor id 605 parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/065861c3-2b9a-4f85-9eb7-d1795ecdbbf1_characterizeImage_1124828_12.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/065861c3-2b9a-4f85-9eb7-d1795ecdbbf1_characterizeImage_1124828_12.stderr parsl.dataflow.dflow INFO: Standard output for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/37aaf610-18a5-4cdd-a185-b43016359892_isr_1124828_12.stdout parsl.dataflow.dflow INFO: Standard error for task 876 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/37aaf610-18a5-4cdd-a185-b43016359892_isr_1124828_12.stderr parsl.dataflow.dflow INFO: Task 879 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e92cdb14f524aa9f7d18410a1e87e44b with result from task 879 parsl.dataflow.dflow DEBUG: Task 879 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 880 has memoization hash 2724e7adeb015e5dced01410d7acc428 parsl.dataflow.memoization INFO: Task 880 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960ade0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 880 try 0 launched on executor multi with executor id 606 parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7f9ded49-ad0f-41ba-bf41-ace6f7d599f7_characterizeImage_1124704_54.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7f9ded49-ad0f-41ba-bf41-ace6f7d599f7_characterizeImage_1124704_54.stderr parsl.dataflow.dflow INFO: Standard output for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/05d6d796-05b1-4997-bce8-8064a3cf8ace_isr_1124704_54.stdout parsl.dataflow.dflow INFO: Standard error for task 879 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/05d6d796-05b1-4997-bce8-8064a3cf8ace_isr_1124704_54.stderr parsl.dataflow.dflow INFO: Task 915 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44ec87e3639357961f3bc9b0339dadc3 with result from task 915 parsl.dataflow.dflow DEBUG: Task 915 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 916 has memoization hash 18120ba72e0bf9793414613fd666429b parsl.dataflow.memoization INFO: Task 916 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529664c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 916 try 0 launched on executor multi with executor id 607 parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/67676cf3-fd81-462b-97bc-07793e84167a_characterizeImage_1124828_24.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/67676cf3-fd81-462b-97bc-07793e84167a_characterizeImage_1124828_24.stderr parsl.dataflow.dflow INFO: Standard output for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/c0203007-7011-4175-a3ca-c9ce219bebff_isr_1124828_24.stdout parsl.dataflow.dflow INFO: Standard error for task 915 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/c0203007-7011-4175-a3ca-c9ce219bebff_isr_1124828_24.stderr parsl.dataflow.dflow INFO: Task 900 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a44b5ec5f8dd4d9c413bfbbae8f043e7 with result from task 900 parsl.dataflow.dflow DEBUG: Task 900 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 901 has memoization hash 8bcc9ee5066093ebe9ad1a136b9ddf1e parsl.dataflow.memoization INFO: Task 901 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c96c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 901 try 0 launched on executor multi with executor id 608 parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/46eb9269-cc5f-431c-9006-8d923e856060_characterizeImage_1124704_23.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/46eb9269-cc5f-431c-9006-8d923e856060_characterizeImage_1124704_23.stderr parsl.dataflow.dflow INFO: Standard output for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/47612486-9fdd-4376-85f2-7cef0c982bfa_isr_1124704_23.stdout parsl.dataflow.dflow INFO: Standard error for task 900 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/47612486-9fdd-4376-85f2-7cef0c982bfa_isr_1124704_23.stderr parsl.dataflow.dflow INFO: Task 894 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc043fc84ee59d02fdc0aa0d89b49e57 with result from task 894 parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:34 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow DEBUG: Task 894 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 895 has memoization hash 3a33da76d4a48aeb4d0095884081e502 parsl.dataflow.memoization INFO: Task 895 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960bd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 895 try 0 launched on executor multi with executor id 609 parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d1071e99-4e87-48c9-b246-90a10057faaf_characterizeImage_1124710_59.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d1071e99-4e87-48c9-b246-90a10057faaf_characterizeImage_1124710_59.stderr parsl.dataflow.dflow INFO: Standard output for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/2ffa1e49-dda0-4148-a2f4-b68186242566_isr_1124710_59.stdout parsl.dataflow.dflow INFO: Standard error for task 894 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/2ffa1e49-dda0-4148-a2f4-b68186242566_isr_1124710_59.stderr parsl.dataflow.dflow INFO: Task 906 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf62743aef00a4879a0e24df5f13ca5c with result from task 906 parsl.dataflow.dflow DEBUG: Task 906 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 907 has memoization hash 4ce8bc4ce82ef946192cce3472b00655 parsl.dataflow.memoization INFO: Task 907 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296674c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 907 try 0 launched on executor multi with executor id 610 parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e1a23abc-5cb4-4ce5-a9cd-4dbf1d761404_characterizeImage_1124707_50.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e1a23abc-5cb4-4ce5-a9cd-4dbf1d761404_characterizeImage_1124707_50.stderr parsl.dataflow.dflow INFO: Standard output for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/f4466f56-8e1e-4333-b78b-0711f119b5f4_isr_1124707_50.stdout parsl.dataflow.dflow INFO: Standard error for task 906 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/f4466f56-8e1e-4333-b78b-0711f119b5f4_isr_1124707_50.stderr parsl.dataflow.dflow INFO: Task 912 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ce1be39c21bf350e349716ebfc7d6dae with result from task 912 parsl.dataflow.dflow DEBUG: Task 912 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 913 has memoization hash e6890062b8db0195580eae54026c9866 parsl.dataflow.memoization INFO: Task 913 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529664900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 913 try 0 launched on executor multi with executor id 611 parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/53a1fe1d-cd0d-4002-9270-9c244ac4fba6_characterizeImage_1124710_57.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/53a1fe1d-cd0d-4002-9270-9c244ac4fba6_characterizeImage_1124710_57.stderr parsl.dataflow.dflow INFO: Standard output for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/498b6c24-b505-4ee9-b29d-8b03a07ba074_isr_1124710_57.stdout parsl.dataflow.dflow INFO: Standard error for task 912 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/498b6c24-b505-4ee9-b29d-8b03a07ba074_isr_1124710_57.stderr parsl.dataflow.dflow INFO: Task 903 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 21d2e71d5619128a479bd44dade809cb with result from task 903 parsl.dataflow.dflow DEBUG: Task 903 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 904 has memoization hash e0bca5306f5515e9f4843afc481698d1 parsl.dataflow.memoization INFO: Task 904 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529667880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 904 try 0 launched on executor multi with executor id 612 parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8211d449-7a80-4243-9356-6c3a823807e4_characterizeImage_1124707_1.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8211d449-7a80-4243-9356-6c3a823807e4_characterizeImage_1124707_1.stderr parsl.dataflow.dflow INFO: Standard output for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/d6fcb666-0a91-4534-8bd5-c5e13da00de7_isr_1124707_1.stdout parsl.dataflow.dflow INFO: Standard error for task 903 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/d6fcb666-0a91-4534-8bd5-c5e13da00de7_isr_1124707_1.stderr parsl.dataflow.dflow INFO: Task 909 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 144bc66aed8ad9364a2893c20528ad36 with result from task 909 parsl.dataflow.dflow DEBUG: Task 909 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 910 has memoization hash e0f20a68fb9ba4fa902a31595d51be66 parsl.dataflow.memoization INFO: Task 910 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296671a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 910 try 0 launched on executor multi with executor id 613 parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/a2aadef0-9a6c-4c0d-a247-31296b9cd1c3_characterizeImage_1124710_61.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/a2aadef0-9a6c-4c0d-a247-31296b9cd1c3_characterizeImage_1124710_61.stderr parsl.dataflow.dflow INFO: Standard output for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/bea03b83-2902-4677-bdd9-8392fcd153e2_isr_1124710_61.stdout parsl.dataflow.dflow INFO: Standard error for task 909 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/bea03b83-2902-4677-bdd9-8392fcd153e2_isr_1124710_61.stderr parsl.dataflow.dflow INFO: Task 897 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c4f5537bcf316765cb14f4d26574fe2 with result from task 897 parsl.dataflow.dflow DEBUG: Task 897 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 898 has memoization hash 952f1537eabc0300c3b4c0ba0c84c6e0 parsl.dataflow.memoization INFO: Task 898 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 898 try 0 launched on executor multi with executor id 614 parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/4108c9f4-94af-4b52-8374-e16303d63664_characterizeImage_1124704_44.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/4108c9f4-94af-4b52-8374-e16303d63664_characterizeImage_1124704_44.stderr parsl.dataflow.dflow INFO: Standard output for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/500bc128-6b2d-4af7-8652-1b9acc85bfe6_isr_1124704_44.stdout parsl.dataflow.dflow INFO: Standard error for task 897 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/500bc128-6b2d-4af7-8652-1b9acc85bfe6_isr_1124704_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:35 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:36 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 708 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cdec8f8a7d1393ca9336f726193b679 with result from task 708 parsl.dataflow.dflow DEBUG: Task 708 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 709 has memoization hash cd2f30ada4da8fa7180068b3e534711f parsl.dataflow.memoization INFO: Task 709 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973ae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 709 try 0 launched on executor multi with executor id 615 parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/5c6b30f2-e105-4222-abb7-3c9554b7fd8a_characterizeImage_1124707_33.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/5c6b30f2-e105-4222-abb7-3c9554b7fd8a_characterizeImage_1124707_33.stderr parsl.dataflow.dflow INFO: Standard output for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/242ebfef-de11-4667-81c6-137b91de8a84_isr_1124707_33.stdout parsl.dataflow.dflow INFO: Standard error for task 708 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/242ebfef-de11-4667-81c6-137b91de8a84_isr_1124707_33.stderr parsl.dataflow.dflow INFO: Task 705 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f2b837d735b808a579b759f6536e61e8 with result from task 705 parsl.dataflow.dflow DEBUG: Task 705 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 706 has memoization hash c61958160d0137d7d253ba0cd334aed7 parsl.dataflow.memoization INFO: Task 706 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973b1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 706 try 0 launched on executor multi with executor id 616 parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/83d39810-cc60-414d-81ea-a3c2eb390752_characterizeImage_1124831_15.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/83d39810-cc60-414d-81ea-a3c2eb390752_characterizeImage_1124831_15.stderr parsl.dataflow.dflow INFO: Standard output for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/034fad09-72f7-4399-bed2-c264726730f8_isr_1124831_15.stdout parsl.dataflow.dflow INFO: Standard error for task 705 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124831/034fad09-72f7-4399-bed2-c264726730f8_isr_1124831_15.stderr parsl.dataflow.dflow INFO: Task 702 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6065334344f9f6b7338da23af151d209 with result from task 702 parsl.dataflow.dflow DEBUG: Task 702 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 703 has memoization hash e6ace7840b88ca7ab44b5900b8cd83de parsl.dataflow.memoization INFO: Task 703 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973b4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 703 try 0 launched on executor multi with executor id 617 parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a9a96111-ea1a-4196-879c-508c571bcdfa_characterizeImage_1124828_19.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a9a96111-ea1a-4196-879c-508c571bcdfa_characterizeImage_1124828_19.stderr parsl.dataflow.dflow INFO: Standard output for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/756b1f54-ddce-4c0a-9ce1-89c859fedd4e_isr_1124828_19.stdout parsl.dataflow.dflow INFO: Standard error for task 702 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124828/756b1f54-ddce-4c0a-9ce1-89c859fedd4e_isr_1124828_19.stderr parsl.dataflow.dflow INFO: Task 717 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b49b2907459030b6c5d140f9f4f46eb with result from task 717 parsl.dataflow.dflow DEBUG: Task 717 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 718 has memoization hash 35f0dc95bb387774e63407ae33e3a36e parsl.dataflow.memoization INFO: Task 718 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 718 try 0 launched on executor multi with executor id 618 parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/fa5cbde5-8361-4567-8314-b66bcec04cd3_characterizeImage_1124707_13.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/fa5cbde5-8361-4567-8314-b66bcec04cd3_characterizeImage_1124707_13.stderr parsl.dataflow.dflow INFO: Standard output for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c9a38688-1deb-4a36-a3e2-2a338d18bc0c_isr_1124707_13.stdout parsl.dataflow.dflow INFO: Standard error for task 717 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/c9a38688-1deb-4a36-a3e2-2a338d18bc0c_isr_1124707_13.stderr parsl.dataflow.dflow INFO: Task 711 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 75b0b7321ebe1d8e0f33a15daef73b17 with result from task 711 parsl.dataflow.dflow DEBUG: Task 711 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 712 has memoization hash 3fa795a8bed149f19cd8c9c8de83af50 parsl.dataflow.memoization INFO: Task 712 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 712 try 0 launched on executor multi with executor id 619 parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/79de0100-d2d0-4bc0-97ea-eafd4449ff6b_characterizeImage_1124710_40.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/79de0100-d2d0-4bc0-97ea-eafd4449ff6b_characterizeImage_1124710_40.stderr parsl.dataflow.dflow INFO: Standard output for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a71f3239-3551-4270-943e-39907281ba25_isr_1124710_40.stdout parsl.dataflow.dflow INFO: Standard error for task 711 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124710/a71f3239-3551-4270-943e-39907281ba25_isr_1124710_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:37 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 720 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eec1c03ed57eda671a6e860237552f39 with result from task 720 parsl.dataflow.dflow DEBUG: Task 720 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 721 has memoization hash 89e08732052415b6815cd57395ddc430 parsl.dataflow.memoization INFO: Task 721 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 721 try 0 launched on executor multi with executor id 620 parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/26c51cd6-e4e5-46e9-abb2-edcf745b6574_characterizeImage_1124707_44.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/26c51cd6-e4e5-46e9-abb2-edcf745b6574_characterizeImage_1124707_44.stderr parsl.dataflow.dflow INFO: Standard output for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/fd5430af-f5d7-4c1c-bc24-6ed09eb85cda_isr_1124707_44.stdout parsl.dataflow.dflow INFO: Standard error for task 720 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124707/fd5430af-f5d7-4c1c-bc24-6ed09eb85cda_isr_1124707_44.stderr parsl.dataflow.dflow INFO: Task 699 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 853ae586cb7b918bb61da76dcee7b57e with result from task 699 parsl.dataflow.dflow DEBUG: Task 699 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 700 has memoization hash 6beaafd98a2fdac448778b00bddfe22b parsl.dataflow.memoization INFO: Task 700 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973b7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 700 try 0 launched on executor multi with executor id 621 parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/641756cc-bb83-4115-abf7-adfe78b7454c_characterizeImage_1124704_16.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/641756cc-bb83-4115-abf7-adfe78b7454c_characterizeImage_1124704_16.stderr parsl.dataflow.dflow INFO: Standard output for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6b1e237f-01f4-42e0-b6e9-3bd2a7a7d69c_isr_1124704_16.stdout parsl.dataflow.dflow INFO: Standard error for task 699 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/6b1e237f-01f4-42e0-b6e9-3bd2a7a7d69c_isr_1124704_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:38 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 714 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 77c8aa7e3fab818a49ea10572579964d with result from task 714 parsl.dataflow.dflow DEBUG: Task 714 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 715 has memoization hash c8b825bc5a592437df6dfb13c873b017 parsl.dataflow.memoization INFO: Task 715 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973bf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 715 try 0 launched on executor multi with executor id 622 parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/e564d309-f127-416c-b404-961bef2a7de9_characterizeImage_1124704_61.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/e564d309-f127-416c-b404-961bef2a7de9_characterizeImage_1124704_61.stderr parsl.dataflow.dflow INFO: Standard output for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e8e468f7-575c-46ee-9b19-de61627c7e37_isr_1124704_61.stdout parsl.dataflow.dflow INFO: Standard error for task 714 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/isr/1124704/e8e468f7-575c-46ee-9b19-de61627c7e37_isr_1124704_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:39 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 100 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de714c66f20a9a4fec89b5cefaf6892c with result from task 100 parsl.dataflow.dflow DEBUG: Task 100 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 101 has memoization hash 264aa7c36c9ef9627a379788d67bc1b2 parsl.dataflow.memoization INFO: Task 101 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 101 try 0 launched on executor multi with executor id 623 parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/ea400acc-1cfe-4054-ac8a-b92e473e398e_calibrate_1124831_27.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/ea400acc-1cfe-4054-ac8a-b92e473e398e_calibrate_1124831_27.stderr parsl.dataflow.dflow INFO: Standard output for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/bff5f210-e9f3-45ce-b6e6-731209fce737_characterizeImage_1124831_27.stdout parsl.dataflow.dflow INFO: Standard error for task 100 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/bff5f210-e9f3-45ce-b6e6-731209fce737_characterizeImage_1124831_27.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 733 try 0 failed parsl.dataflow.dflow INFO: Task 733 marked for retry parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/23f01ed6-d153-43be-8984-62e5f76efe73_characterizeImage_1124710_2.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/23f01ed6-d153-43be-8984-62e5f76efe73_characterizeImage_1124710_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 733 has memoization hash 0e6631dc1017587635539c370f059183 parsl.dataflow.memoization INFO: Task 733 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 733 try 1 launched on executor multi with executor id 624 parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/23f01ed6-d153-43be-8984-62e5f76efe73_characterizeImage_1124710_2.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/23f01ed6-d153-43be-8984-62e5f76efe73_characterizeImage_1124710_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:40 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 370 try 0 failed parsl.dataflow.dflow INFO: Task 370 marked for retry parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/45b82ec0-bfba-47fd-baa4-5f7cbe3dceab_characterizeImage_1124831_61.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/45b82ec0-bfba-47fd-baa4-5f7cbe3dceab_characterizeImage_1124831_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 370 has memoization hash f2e3ec23c7297a32110a9ffdedcd1967 parsl.dataflow.memoization INFO: Task 370 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 370 try 1 launched on executor multi with executor id 625 parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/45b82ec0-bfba-47fd-baa4-5f7cbe3dceab_characterizeImage_1124831_61.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/45b82ec0-bfba-47fd-baa4-5f7cbe3dceab_characterizeImage_1124831_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:41 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 592 try 0 failed parsl.dataflow.dflow INFO: Task 592 marked for retry parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3464ff69-e292-4267-b1de-41e6fa27387c_characterizeImage_1124707_2.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3464ff69-e292-4267-b1de-41e6fa27387c_characterizeImage_1124707_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 592 has memoization hash fecb4c03f58e9e88baba6e363c229ef3 parsl.dataflow.memoization INFO: Task 592 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969fb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 592 try 1 launched on executor multi with executor id 626 parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3464ff69-e292-4267-b1de-41e6fa27387c_characterizeImage_1124707_2.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3464ff69-e292-4267-b1de-41e6fa27387c_characterizeImage_1124707_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:42 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:43 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:44 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:45 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:46 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:47 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:48 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 10 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d745c990f8bab85b09cd0cc4a3f2288 with result from task 10 parsl.dataflow.dflow DEBUG: Task 10 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 11 has memoization hash 6b695129f77563daad338b9ae5819cde parsl.dataflow.memoization INFO: Task 11 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad49800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 11 try 0 launched on executor multi with executor id 627 parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/b8e4b587-45d1-40ca-8330-81fb95068ca6_calibrate_1124831_5.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/b8e4b587-45d1-40ca-8330-81fb95068ca6_calibrate_1124831_5.stderr parsl.dataflow.dflow INFO: Standard output for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/e6d1ac57-a151-401d-8b2a-aa6bc3130491_characterizeImage_1124831_5.stdout parsl.dataflow.dflow INFO: Standard error for task 10 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/e6d1ac57-a151-401d-8b2a-aa6bc3130491_characterizeImage_1124831_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:49 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:50 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:51 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 322 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14b5083b0774bf7d242c0823a31afcae with result from task 322 parsl.dataflow.dflow DEBUG: Task 322 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 323 has memoization hash a8e186211c167845d36a83300f217d6a parsl.dataflow.memoization INFO: Task 323 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ce0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 323 try 0 launched on executor multi with executor id 628 parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/e7f93319-d94d-48f7-afa8-acfb5acfebe0_calibrate_1124831_36.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/e7f93319-d94d-48f7-afa8-acfb5acfebe0_calibrate_1124831_36.stderr parsl.dataflow.dflow INFO: Standard output for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/0ed03dd4-3343-4e65-b1b0-ce8e1ae9bef2_characterizeImage_1124831_36.stdout parsl.dataflow.dflow INFO: Standard error for task 322 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/0ed03dd4-3343-4e65-b1b0-ce8e1ae9bef2_characterizeImage_1124831_36.stderr parsl.dataflow.dflow INFO: Task 304 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4c0a5cc455347cd2e2a93a2e4efeb549 with result from task 304 parsl.dataflow.dflow DEBUG: Task 304 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 305 has memoization hash 44dc80a8f45b12843c30de8048a5a913 parsl.dataflow.memoization INFO: Task 305 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cce00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 305 try 0 launched on executor multi with executor id 629 parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/9ba2305e-e42f-498a-824b-a9750c519986_calibrate_1124831_14.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/9ba2305e-e42f-498a-824b-a9750c519986_calibrate_1124831_14.stderr parsl.dataflow.dflow INFO: Standard output for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/a614cf75-b770-4492-9dd4-19da8401e4a4_characterizeImage_1124831_14.stdout parsl.dataflow.dflow INFO: Standard error for task 304 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/a614cf75-b770-4492-9dd4-19da8401e4a4_characterizeImage_1124831_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:52 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 310 parsl.dataflow.strategy DEBUG: Executor multi has 310 active tasks, 72/6 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:27:53 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:27:54 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 355 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d3f6b7e227a0e9f5567582c9598235af with result from task 355 parsl.dataflow.dflow DEBUG: Task 355 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 356 has memoization hash 62cad87d931b9e1a023faf9c35445082 parsl.dataflow.memoization INFO: Task 356 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 356 try 0 launched on executor multi with executor id 630 parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/18892a54-5eb0-4555-a1a1-1b8d1420b5a6_calibrate_1124828_29.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/18892a54-5eb0-4555-a1a1-1b8d1420b5a6_calibrate_1124828_29.stderr parsl.dataflow.dflow INFO: Standard output for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/fe2a1a13-b65a-4225-a5d7-5ac6fc728611_characterizeImage_1124828_29.stdout parsl.dataflow.dflow INFO: Standard error for task 355 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/fe2a1a13-b65a-4225-a5d7-5ac6fc728611_characterizeImage_1124828_29.stderr parsl.dataflow.dflow INFO: Task 268 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63af4649a3bd860917824739eac1e6c9 with result from task 268 parsl.dataflow.dflow DEBUG: Task 268 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 269 has memoization hash 2070557cf246ffc33f3b08b3c115102f parsl.dataflow.memoization INFO: Task 269 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aa840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 269 try 0 launched on executor multi with executor id 631 parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3fbcd8a9-b45c-4227-83a3-77c1b03d97de_calibrate_1124831_19.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3fbcd8a9-b45c-4227-83a3-77c1b03d97de_calibrate_1124831_19.stderr parsl.dataflow.dflow INFO: Standard output for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/ada01af9-6351-4eb9-a3b5-3191672af0be_characterizeImage_1124831_19.stdout parsl.dataflow.dflow INFO: Standard error for task 268 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/ada01af9-6351-4eb9-a3b5-3191672af0be_characterizeImage_1124831_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:55 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 70 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fb7f156f8dcacead2f02ea3139da5e9 with result from task 70 parsl.dataflow.dflow DEBUG: Task 70 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 71 has memoization hash f40f186add58351ad52e6f8902e9a02e parsl.dataflow.memoization INFO: Task 71 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad65940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 71 try 0 launched on executor multi with executor id 632 parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/45ca6e14-e625-42fe-b653-ff41074de4d7_calibrate_1124831_47.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/45ca6e14-e625-42fe-b653-ff41074de4d7_calibrate_1124831_47.stderr parsl.dataflow.dflow INFO: Standard output for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/2a8aebfe-2697-4fe1-8621-6c3552a2bf72_characterizeImage_1124831_47.stdout parsl.dataflow.dflow INFO: Standard error for task 70 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/2a8aebfe-2697-4fe1-8621-6c3552a2bf72_characterizeImage_1124831_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:56 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:57 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 82 try 0 failed parsl.dataflow.dflow INFO: Task 82 marked for retry parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/7021b40b-e213-4f60-b138-ffdbfbf58281_characterizeImage_1124828_61.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/7021b40b-e213-4f60-b138-ffdbfbf58281_characterizeImage_1124828_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 82 has memoization hash 1dbe3e95e947bbde0227f98a31162e9f parsl.dataflow.memoization INFO: Task 82 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad667a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 82 try 1 launched on executor multi with executor id 633 parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/7021b40b-e213-4f60-b138-ffdbfbf58281_characterizeImage_1124828_61.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/7021b40b-e213-4f60-b138-ffdbfbf58281_characterizeImage_1124828_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:58 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 127 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 557b6d69231c25a3cd860b3078678df5 with result from task 127 parsl.dataflow.dflow DEBUG: Task 127 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 128 has memoization hash bef96d85909e943687eaea34ed95c5f5 parsl.dataflow.memoization INFO: Task 128 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8d4e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 128 try 0 launched on executor multi with executor id 634 parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/379f35c8-0a2e-4ca3-ad38-bc02f7d19b7c_calibrate_1124828_13.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/379f35c8-0a2e-4ca3-ad38-bc02f7d19b7c_calibrate_1124828_13.stderr parsl.dataflow.dflow INFO: Standard output for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/9a0f653e-dfdb-4c03-b69f-c50e4c6574fb_characterizeImage_1124828_13.stdout parsl.dataflow.dflow INFO: Standard error for task 127 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/9a0f653e-dfdb-4c03-b69f-c50e4c6574fb_characterizeImage_1124828_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:27:59 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 313 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7da54d1fb39df461903f51d053b3d85b with result from task 313 parsl.dataflow.dflow DEBUG: Task 313 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 314 has memoization hash 7a506dd989d7bbe3e77919ef1f4ebb96 parsl.dataflow.memoization INFO: Task 314 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cd760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 314 try 0 launched on executor multi with executor id 635 parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3bac62ac-b48f-4bba-a73d-cad60cbb8c0e_calibrate_1124831_1.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3bac62ac-b48f-4bba-a73d-cad60cbb8c0e_calibrate_1124831_1.stderr parsl.dataflow.dflow INFO: Standard output for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f3ccc0fd-66ac-44dc-8b24-982d8c05ea72_characterizeImage_1124831_1.stdout parsl.dataflow.dflow INFO: Standard error for task 313 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f3ccc0fd-66ac-44dc-8b24-982d8c05ea72_characterizeImage_1124831_1.stderr parsl.dataflow.dflow INFO: Task 67 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc6099b4e37a2bb55c7675e3936825af with result from task 67 parsl.dataflow.dflow DEBUG: Task 67 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 68 has memoization hash b79270740dc1be70f6950f556135ac88 parsl.dataflow.memoization INFO: Task 68 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad65620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 68 try 0 launched on executor multi with executor id 636 parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f55bd7e2-76bb-4368-9a07-09af0bf3b95c_calibrate_1124828_59.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f55bd7e2-76bb-4368-9a07-09af0bf3b95c_calibrate_1124828_59.stderr parsl.dataflow.dflow INFO: Standard output for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/dbed9bce-2aa8-44fb-bd40-454657e2ade2_characterizeImage_1124828_59.stdout parsl.dataflow.dflow INFO: Standard error for task 67 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/dbed9bce-2aa8-44fb-bd40-454657e2ade2_characterizeImage_1124828_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:00 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:01 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 46 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 99b96d27c484ad42f0277a342757ab20 with result from task 46 parsl.dataflow.dflow DEBUG: Task 46 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 47 has memoization hash bcd35d136d0f9dad871b9c448f5fcedd parsl.dataflow.memoization INFO: Task 47 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 47 try 0 launched on executor multi with executor id 637 parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/cb3db7b8-caa3-438a-bcd3-91429c967614_calibrate_1124831_33.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/cb3db7b8-caa3-438a-bcd3-91429c967614_calibrate_1124831_33.stderr parsl.dataflow.dflow INFO: Standard output for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/724b3603-462f-47fa-8a08-8976c27db3d4_characterizeImage_1124831_33.stdout parsl.dataflow.dflow INFO: Standard error for task 46 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/724b3603-462f-47fa-8a08-8976c27db3d4_characterizeImage_1124831_33.stderr parsl.dataflow.dflow INFO: Task 427 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 999a8be926a22baba70afcd019c688d8 with result from task 427 parsl.dataflow.dflow DEBUG: Task 427 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 428 has memoization hash dbf96f9cc5aac16b976eadbd1a49ab11 parsl.dataflow.memoization INFO: Task 428 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981c7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 428 try 0 launched on executor multi with executor id 638 parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/1a37c963-4a78-43af-9394-f19568fa7c20_calibrate_1124831_60.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/1a37c963-4a78-43af-9394-f19568fa7c20_calibrate_1124831_60.stderr parsl.dataflow.dflow INFO: Standard output for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/d621ef3f-f21a-4526-906c-927a96a1de6a_characterizeImage_1124831_60.stdout parsl.dataflow.dflow INFO: Standard error for task 427 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/d621ef3f-f21a-4526-906c-927a96a1de6a_characterizeImage_1124831_60.stderr parsl.dataflow.dflow INFO: Task 421 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 10444017c6fcdfee44e03cfe8e0dd470 with result from task 421 parsl.dataflow.dflow DEBUG: Task 421 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 422 has memoization hash d06068846ac7cc85b7f099daf34b8b8f parsl.dataflow.memoization INFO: Task 422 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 422 try 0 launched on executor multi with executor id 639 parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f56f3e96-f883-46e6-a55d-e4e175ea609c_calibrate_1124828_16.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f56f3e96-f883-46e6-a55d-e4e175ea609c_calibrate_1124828_16.stderr parsl.dataflow.dflow INFO: Standard output for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/49857fee-66ca-41f3-9d0b-ab61610a1738_characterizeImage_1124828_16.stdout parsl.dataflow.dflow INFO: Standard error for task 421 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/49857fee-66ca-41f3-9d0b-ab61610a1738_characterizeImage_1124828_16.stderr parsl.dataflow.dflow INFO: Task 211 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09cceba7ef785e512ab711a01a6576dd with result from task 211 parsl.dataflow.dflow DEBUG: Task 211 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 212 has memoization hash 90a5dca4a4e91ce7f68f3fa7cbde2ca4 parsl.dataflow.memoization INFO: Task 212 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaeca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 212 try 0 launched on executor multi with executor id 640 parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8823aba8-c4c5-4c70-8b0e-c030e2c216d3_calibrate_1124828_9.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8823aba8-c4c5-4c70-8b0e-c030e2c216d3_calibrate_1124828_9.stderr parsl.dataflow.dflow INFO: Standard output for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/dbd3cbdb-d947-4c57-9b6d-5c7af12fe27c_characterizeImage_1124828_9.stdout parsl.dataflow.dflow INFO: Standard error for task 211 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/dbd3cbdb-d947-4c57-9b6d-5c7af12fe27c_characterizeImage_1124828_9.stderr parsl.dataflow.dflow INFO: Task 76 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad04feacbb4f2738e67fdc7788d34f82 with result from task 76 parsl.dataflow.dflow DEBUG: Task 76 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 77 has memoization hash 9b225a98483049dee3a3cf1afd82e35e parsl.dataflow.memoization INFO: Task 77 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad65f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 77 try 0 launched on executor multi with executor id 641 parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/fb86883b-98f7-4453-9732-7a4e9ab44ec7_calibrate_1124831_30.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/fb86883b-98f7-4453-9732-7a4e9ab44ec7_calibrate_1124831_30.stderr parsl.dataflow.dflow INFO: Standard output for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f4827b79-86c5-4aa4-8038-49f41b460da3_characterizeImage_1124831_30.stdout parsl.dataflow.dflow INFO: Standard error for task 76 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f4827b79-86c5-4aa4-8038-49f41b460da3_characterizeImage_1124831_30.stderr parsl.dataflow.dflow INFO: Task 787 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d6deabfeaaf9165ed93012871273a6f8 with result from task 787 parsl.dataflow.dflow DEBUG: Task 787 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 788 has memoization hash d13d456aa913c9bfef06fa675d4481cd parsl.dataflow.memoization INFO: Task 788 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c8ae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 788 try 0 launched on executor multi with executor id 642 parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f9b21050-79f3-47e0-ad78-23f149f6e6cd_calibrate_1124828_5.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f9b21050-79f3-47e0-ad78-23f149f6e6cd_calibrate_1124828_5.stderr parsl.dataflow.dflow INFO: Standard output for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/93f38e2d-acf6-4dca-90c6-5443e281453c_characterizeImage_1124828_5.stdout parsl.dataflow.dflow INFO: Standard error for task 787 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/93f38e2d-acf6-4dca-90c6-5443e281453c_characterizeImage_1124828_5.stderr parsl.dataflow.dflow INFO: Task 1 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f434be0ed1ae98f7750bbe444c4bc59 with result from task 1 parsl.dataflow.dflow DEBUG: Task 1 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 2 has memoization hash b6993186717cf4791c4aa41deee6b3f8 parsl.dataflow.memoization INFO: Task 2 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad48ea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 2 try 0 launched on executor multi with executor id 643 parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/38db5b39-d87d-4213-8686-09bce0132a2a_calibrate_1124831_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/38db5b39-d87d-4213-8686-09bce0132a2a_calibrate_1124831_45.stderr parsl.dataflow.dflow INFO: Standard output for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/4be1f4be-c0ab-42e2-b1b1-2c302ab4d21a_characterizeImage_1124831_45.stdout parsl.dataflow.dflow INFO: Standard error for task 1 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/4be1f4be-c0ab-42e2-b1b1-2c302ab4d21a_characterizeImage_1124831_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:04 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 439 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d360466ad3ec7520e08b1b98cfa7a967 with result from task 439 parsl.dataflow.dflow DEBUG: Task 439 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 440 has memoization hash 6a29feffff33908d06ecaa664efb507e parsl.dataflow.memoization INFO: Task 440 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981dee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 440 try 0 launched on executor multi with executor id 644 parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/df3eb3dc-5bb5-4541-8c28-c95ba73ffa96_calibrate_1124828_23.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/df3eb3dc-5bb5-4541-8c28-c95ba73ffa96_calibrate_1124828_23.stderr parsl.dataflow.dflow INFO: Standard output for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/b0d26e0e-ef4c-4625-aedf-57b26b21562c_characterizeImage_1124828_23.stdout parsl.dataflow.dflow INFO: Standard error for task 439 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/b0d26e0e-ef4c-4625-aedf-57b26b21562c_characterizeImage_1124828_23.stderr parsl.dataflow.dflow INFO: Task 559 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4dfbf2a8e1793fd2c2540f011496bb06 with result from task 559 parsl.dataflow.dflow DEBUG: Task 559 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 560 has memoization hash 8758af19badc1dc49176f87032743792 parsl.dataflow.memoization INFO: Task 560 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 560 try 0 launched on executor multi with executor id 645 parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a1cf385e-d375-4f93-b8c8-c0e0a92888c0_calibrate_1124831_9.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a1cf385e-d375-4f93-b8c8-c0e0a92888c0_calibrate_1124831_9.stderr parsl.dataflow.dflow INFO: Standard output for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5d56936e-b801-4ebc-9a00-4d05a0ce7b15_characterizeImage_1124831_9.stdout parsl.dataflow.dflow INFO: Standard error for task 559 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5d56936e-b801-4ebc-9a00-4d05a0ce7b15_characterizeImage_1124831_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:06 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:07 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 61 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b05359cc98cbdea4ffe6ce5031ac5b9 with result from task 61 parsl.dataflow.dflow DEBUG: Task 61 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 62 has memoization hash 010385f0374f87e56bbd2ad99afb64eb parsl.dataflow.memoization INFO: Task 62 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 62 try 0 launched on executor multi with executor id 646 parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/699a6afc-cec4-4df9-a07b-dbe11fdfd89f_calibrate_1124831_40.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/699a6afc-cec4-4df9-a07b-dbe11fdfd89f_calibrate_1124831_40.stderr parsl.dataflow.dflow INFO: Standard output for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/a2424d05-6fbc-42ea-b66f-698bd46e9f52_characterizeImage_1124831_40.stdout parsl.dataflow.dflow INFO: Standard error for task 61 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/a2424d05-6fbc-42ea-b66f-698bd46e9f52_characterizeImage_1124831_40.stderr parsl.dataflow.dflow INFO: Task 109 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69fefaa500fdf1caf5c71740baf766e2 with result from task 109 parsl.dataflow.dflow DEBUG: Task 109 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 110 has memoization hash a8a1faaf4b0b4958f3b5c0061e36d6aa parsl.dataflow.memoization INFO: Task 110 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8c220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 110 try 0 launched on executor multi with executor id 647 parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/b3530352-5e91-4a44-b83e-faff8d7d98dd_calibrate_1124831_42.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/b3530352-5e91-4a44-b83e-faff8d7d98dd_calibrate_1124831_42.stderr parsl.dataflow.dflow INFO: Standard output for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/968ea4fb-80eb-4249-9c08-99297ed07fef_characterizeImage_1124831_42.stdout parsl.dataflow.dflow INFO: Standard error for task 109 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/968ea4fb-80eb-4249-9c08-99297ed07fef_characterizeImage_1124831_42.stderr parsl.dataflow.dflow INFO: Task 607 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 03dbf4e6b12f7efe5c32f4a48c63399f with result from task 607 parsl.dataflow.dflow DEBUG: Task 607 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 608 has memoization hash 54ad86dd6135b5ef0387fbcfc0319f34 parsl.dataflow.memoization INFO: Task 608 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e8860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 608 try 0 launched on executor multi with executor id 648 parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/509c125f-07b9-45de-8498-6eeea71bf134_calibrate_1124828_1.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/509c125f-07b9-45de-8498-6eeea71bf134_calibrate_1124828_1.stderr parsl.dataflow.dflow INFO: Standard output for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/d43d61ce-a68a-4b07-9f31-8cc93c1e43f7_characterizeImage_1124828_1.stdout parsl.dataflow.dflow INFO: Standard error for task 607 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/d43d61ce-a68a-4b07-9f31-8cc93c1e43f7_characterizeImage_1124828_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:08 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 373 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d291f89802e37d3b24d5dd61ebe0027 with result from task 373 parsl.dataflow.dflow DEBUG: Task 373 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 374 has memoization hash 0180598a735c7f91db242f93fe811840 parsl.dataflow.memoization INFO: Task 374 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e9620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 374 try 0 launched on executor multi with executor id 649 parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/5e6dd009-02d0-4a75-a867-afcddf19492a_calibrate_1124831_29.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/5e6dd009-02d0-4a75-a867-afcddf19492a_calibrate_1124831_29.stderr parsl.dataflow.dflow INFO: Standard output for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/becbcabc-d5eb-4045-adc2-3f6dabc234e4_characterizeImage_1124831_29.stdout parsl.dataflow.dflow INFO: Standard error for task 373 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/becbcabc-d5eb-4045-adc2-3f6dabc234e4_characterizeImage_1124831_29.stderr parsl.dataflow.dflow INFO: Task 553 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2aee5d75184e117b18f0f20a422dea06 with result from task 553 parsl.dataflow.dflow DEBUG: Task 553 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 554 has memoization hash a8691e1bcc463482942b125b221ada2f parsl.dataflow.memoization INFO: Task 554 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969cc20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 554 try 0 launched on executor multi with executor id 650 parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/5305c0a0-d689-4770-9dbf-fb29774a7a28_calibrate_1124831_8.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/5305c0a0-d689-4770-9dbf-fb29774a7a28_calibrate_1124831_8.stderr parsl.dataflow.dflow INFO: Standard output for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/53362882-773d-4665-847d-5b99f7f01a17_characterizeImage_1124831_8.stdout parsl.dataflow.dflow INFO: Standard error for task 553 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/53362882-773d-4665-847d-5b99f7f01a17_characterizeImage_1124831_8.stderr parsl.dataflow.dflow INFO: Task 337 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1259d88bc3d3a35d904d7fff46167b2f with result from task 337 parsl.dataflow.dflow DEBUG: Task 337 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 338 has memoization hash a3e2897a85cbfe7700255b1e15a1aa2d parsl.dataflow.memoization INFO: Task 338 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cf060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 338 try 0 launched on executor multi with executor id 651 parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/51596cec-fbde-4e06-b45a-545dc377a4fd_calibrate_1124828_31.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/51596cec-fbde-4e06-b45a-545dc377a4fd_calibrate_1124828_31.stderr parsl.dataflow.dflow INFO: Standard output for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/c927bf96-ef83-4ba7-a073-9605897f12d4_characterizeImage_1124828_31.stdout parsl.dataflow.dflow INFO: Standard error for task 337 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/c927bf96-ef83-4ba7-a073-9605897f12d4_characterizeImage_1124828_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:09 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 13 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5925da6d2e4797c08efd15eb600b9d0 with result from task 13 parsl.dataflow.dflow DEBUG: Task 13 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 14 has memoization hash dee48d2ff5223a0d610b73fb4b26c11e parsl.dataflow.memoization INFO: Task 14 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad49a80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 14 try 0 launched on executor multi with executor id 652 parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/6d615dec-b5c8-42ce-8bce-9eb89254df50_calibrate_1124828_28.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/6d615dec-b5c8-42ce-8bce-9eb89254df50_calibrate_1124828_28.stderr parsl.dataflow.dflow INFO: Standard output for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0d9ddfbe-5ff2-410e-9bda-d19ea5688c73_characterizeImage_1124828_28.stdout parsl.dataflow.dflow INFO: Standard error for task 13 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0d9ddfbe-5ff2-410e-9bda-d19ea5688c73_characterizeImage_1124828_28.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 772 try 0 failed parsl.dataflow.dflow INFO: Task 772 marked for retry parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/142f6ce8-f81b-476d-ac52-5acf2bf1c6d5_characterizeImage_1124831_2.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/142f6ce8-f81b-476d-ac52-5acf2bf1c6d5_characterizeImage_1124831_2.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 772 has memoization hash 3de12400f6b8a0bb3edf4a850a7177da parsl.dataflow.memoization INFO: Task 772 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977f1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 772 try 1 launched on executor multi with executor id 653 parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/142f6ce8-f81b-476d-ac52-5acf2bf1c6d5_characterizeImage_1124831_2.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/142f6ce8-f81b-476d-ac52-5acf2bf1c6d5_characterizeImage_1124831_2.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 16 try 1 failed parsl.dataflow.dflow ERROR: Task 16 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 33bb2dcd21588a19fda78b0824addbe2 with result from task 16 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 17 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 17 try 0 failed parsl.dataflow.dflow INFO: Task 17 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8a0749e1-51e3-41ee-aacb-ee2d57a8c6dd_calibrate_1124828_2.stdout parsl.dataflow.dflow INFO: Standard error for task 17 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8a0749e1-51e3-41ee-aacb-ee2d57a8c6dd_calibrate_1124828_2.stderr parsl.dataflow.dflow INFO: Standard output for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/98428587-0f80-48cf-b5ae-ec3269d4888b_characterizeImage_1124828_2.stdout parsl.dataflow.dflow INFO: Standard error for task 16 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/98428587-0f80-48cf-b5ae-ec3269d4888b_characterizeImage_1124828_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:10 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 55 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f4da31e456bc99d6d5d72088d66c7c1 with result from task 55 parsl.dataflow.dflow DEBUG: Task 55 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 56 has memoization hash 25c0c69f928e3f2a0bbe086c88353e44 parsl.dataflow.memoization INFO: Task 56 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad649a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 56 try 0 launched on executor multi with executor id 654 parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/555ce685-4826-4b54-9c51-6291ea540720_calibrate_1124831_6.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/555ce685-4826-4b54-9c51-6291ea540720_calibrate_1124831_6.stderr parsl.dataflow.dflow INFO: Standard output for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/ffd32b93-4043-41ab-8c96-6345a93dee3a_characterizeImage_1124831_6.stdout parsl.dataflow.dflow INFO: Standard error for task 55 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/ffd32b93-4043-41ab-8c96-6345a93dee3a_characterizeImage_1124831_6.stderr parsl.dataflow.dflow INFO: Task 4 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 86b8de94091d790cd4287d0f965bd624 with result from task 4 parsl.dataflow.dflow DEBUG: Task 4 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 5 has memoization hash d55828ffa31ac2ac98223768907b90ec parsl.dataflow.memoization INFO: Task 5 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad49260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 5 try 0 launched on executor multi with executor id 655 parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/66e7bb53-41cd-44b8-ae66-1942aa2b568f_calibrate_1124831_3.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/66e7bb53-41cd-44b8-ae66-1942aa2b568f_calibrate_1124831_3.stderr parsl.dataflow.dflow INFO: Standard output for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/1679141f-8292-4f09-86fe-adbe1ccff61e_characterizeImage_1124831_3.stdout parsl.dataflow.dflow INFO: Standard error for task 4 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/1679141f-8292-4f09-86fe-adbe1ccff61e_characterizeImage_1124831_3.stderr parsl.dataflow.dflow INFO: Task 436 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be37226d2cd7cb27e0defc51d4280e40 with result from task 436 parsl.dataflow.dflow DEBUG: Task 436 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 437 has memoization hash 9193f0f6dfe0c12d6f17af3140e7af98 parsl.dataflow.memoization INFO: Task 437 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 437 try 0 launched on executor multi with executor id 656 parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/fb8f1fc9-fa0c-4609-b6eb-0cfc90a8dddc_calibrate_1124831_44.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/fb8f1fc9-fa0c-4609-b6eb-0cfc90a8dddc_calibrate_1124831_44.stderr parsl.dataflow.dflow INFO: Standard output for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5c7859d8-7cf6-4dc8-90e7-8dc332d53fa2_characterizeImage_1124831_44.stdout parsl.dataflow.dflow INFO: Standard error for task 436 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5c7859d8-7cf6-4dc8-90e7-8dc332d53fa2_characterizeImage_1124831_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:11 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 94 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 31e0f2ff183c9f55140b405c11849a71 with result from task 94 parsl.dataflow.dflow DEBUG: Task 94 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 95 has memoization hash 4ece8d27881a97b860fcc2748aa2a306 parsl.dataflow.memoization INFO: Task 95 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 95 try 0 launched on executor multi with executor id 657 parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/e72563e3-f600-4c51-9122-cd99cf0d95b5_calibrate_1124831_23.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/e72563e3-f600-4c51-9122-cd99cf0d95b5_calibrate_1124831_23.stderr parsl.dataflow.dflow INFO: Standard output for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/39e494c5-8360-41fb-b9e9-fcf12e20f507_characterizeImage_1124831_23.stdout parsl.dataflow.dflow INFO: Standard error for task 94 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/39e494c5-8360-41fb-b9e9-fcf12e20f507_characterizeImage_1124831_23.stderr parsl.dataflow.dflow INFO: Task 40 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06ee2ed5481ce4c190af73a3a94c430a with result from task 40 parsl.dataflow.dflow DEBUG: Task 40 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 41 has memoization hash f482dd6223d62056751d72dc5198e8cd parsl.dataflow.memoization INFO: Task 41 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4a700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 41 try 0 launched on executor multi with executor id 658 parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/33b508ac-790f-4106-a9eb-b611305ad3c7_calibrate_1124828_32.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/33b508ac-790f-4106-a9eb-b611305ad3c7_calibrate_1124828_32.stderr parsl.dataflow.dflow INFO: Standard output for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/17d937da-a8a9-4ccc-8070-ee2f7f2c382f_characterizeImage_1124828_32.stdout parsl.dataflow.dflow INFO: Standard error for task 40 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/17d937da-a8a9-4ccc-8070-ee2f7f2c382f_characterizeImage_1124828_32.stderr parsl.dataflow.dflow INFO: Task 601 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e0c1b00881461c3c006c1b41ee83f1c with result from task 601 parsl.dataflow.dflow DEBUG: Task 601 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 602 has memoization hash 209fefcca37df15a9e0fd224ae1eb26a parsl.dataflow.memoization INFO: Task 602 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 602 try 0 launched on executor multi with executor id 659 parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/37b62b35-237a-46e2-93b2-7c9269b1e41b_calibrate_1124828_10.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/37b62b35-237a-46e2-93b2-7c9269b1e41b_calibrate_1124828_10.stderr parsl.dataflow.dflow INFO: Standard output for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/48c02441-1d48-4b40-8565-b05cfc80db4c_characterizeImage_1124828_10.stdout parsl.dataflow.dflow INFO: Standard error for task 601 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/48c02441-1d48-4b40-8565-b05cfc80db4c_characterizeImage_1124828_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:12 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 583 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 055e6746995c0baa10ae49800fcda7a4 with result from task 583 parsl.dataflow.dflow DEBUG: Task 583 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 584 has memoization hash 50ff061f6e16ab19fe174432e90ac73b parsl.dataflow.memoization INFO: Task 584 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969f240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 584 try 0 launched on executor multi with executor id 660 parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/affd8d00-fe17-4aad-a01f-f52d446c5dc6_calibrate_1124828_54.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/affd8d00-fe17-4aad-a01f-f52d446c5dc6_calibrate_1124828_54.stderr parsl.dataflow.dflow INFO: Standard output for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a2a10ae1-5b06-49d9-b9c1-ce1d04c44c55_characterizeImage_1124828_54.stdout parsl.dataflow.dflow INFO: Standard error for task 583 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a2a10ae1-5b06-49d9-b9c1-ce1d04c44c55_characterizeImage_1124828_54.stderr parsl.dataflow.dflow INFO: Task 328 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38d834b36725c58f3da889e1eedaf116 with result from task 328 parsl.dataflow.dflow DEBUG: Task 328 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 329 has memoization hash b2cfc7d394e521b2ade3c95a4141d866 parsl.dataflow.memoization INFO: Task 329 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ce700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 329 try 0 launched on executor multi with executor id 661 parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/25e47931-91d6-454c-b4a9-e563a3fb3dd9_calibrate_1124828_60.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/25e47931-91d6-454c-b4a9-e563a3fb3dd9_calibrate_1124828_60.stderr parsl.dataflow.dflow INFO: Standard output for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/cdee9122-7d3c-44df-aefa-f3e2b7642429_characterizeImage_1124828_60.stdout parsl.dataflow.dflow INFO: Standard error for task 328 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/cdee9122-7d3c-44df-aefa-f3e2b7642429_characterizeImage_1124828_60.stderr parsl.dataflow.dflow INFO: Task 745 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbadc6fff26178ed3677899f8fdc31a6 with result from task 745 parsl.dataflow.dflow DEBUG: Task 745 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 746 has memoization hash 951234f2eeba18364b48f1bb7926268a parsl.dataflow.memoization INFO: Task 746 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977d580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 746 try 0 launched on executor multi with executor id 662 parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/779b319a-fcd0-4f6e-b949-09c464ab59a1_calibrate_1124831_39.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/779b319a-fcd0-4f6e-b949-09c464ab59a1_calibrate_1124831_39.stderr parsl.dataflow.dflow INFO: Standard output for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5ec2fb67-443f-424e-b7bb-32ff6333c218_characterizeImage_1124831_39.stdout parsl.dataflow.dflow INFO: Standard error for task 745 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5ec2fb67-443f-424e-b7bb-32ff6333c218_characterizeImage_1124831_39.stderr parsl.dataflow.dflow INFO: Task 79 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 247f6d3ba911693bae7e2ef1514049f8 with result from task 79 parsl.dataflow.dflow DEBUG: Task 79 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 80 has memoization hash 8598b0bce161afae7bea16b3d86f94f6 parsl.dataflow.memoization INFO: Task 80 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad662a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 80 try 0 launched on executor multi with executor id 663 parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/38fb9a8a-c24d-419f-8117-c238ae95090b_calibrate_1124710_18.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/38fb9a8a-c24d-419f-8117-c238ae95090b_calibrate_1124710_18.stderr parsl.dataflow.dflow INFO: Standard output for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d0f9ddb7-7029-445c-a2cf-141e39327e6f_characterizeImage_1124710_18.stdout parsl.dataflow.dflow INFO: Standard error for task 79 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d0f9ddb7-7029-445c-a2cf-141e39327e6f_characterizeImage_1124710_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:13 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 49 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2702d1fe70d043d784024b9b808e699 with result from task 49 parsl.dataflow.dflow DEBUG: Task 49 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 50 has memoization hash 0f6af63142a1ae19ee57510b3de3b7b3 parsl.dataflow.memoization INFO: Task 50 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 50 try 0 launched on executor multi with executor id 664 parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/893d01de-ee1d-4bbf-9126-97ece8f1179d_calibrate_1124828_34.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/893d01de-ee1d-4bbf-9126-97ece8f1179d_calibrate_1124828_34.stderr parsl.dataflow.dflow INFO: Standard output for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a2cdd4e2-3bed-4115-ab97-582e59662687_characterizeImage_1124828_34.stdout parsl.dataflow.dflow INFO: Standard error for task 49 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a2cdd4e2-3bed-4115-ab97-582e59662687_characterizeImage_1124828_34.stderr parsl.dataflow.dflow INFO: Task 571 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a5c3f81ed3d5a168d8962acd523bb62 with result from task 571 parsl.dataflow.dflow DEBUG: Task 571 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 572 has memoization hash 362a914c492cd6e4cd9d0b24090fce01 parsl.dataflow.memoization INFO: Task 572 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969d760> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 572 try 0 launched on executor multi with executor id 665 parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/ff773cea-520a-44b1-b70c-5addf5aba004_calibrate_1124831_22.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/ff773cea-520a-44b1-b70c-5addf5aba004_calibrate_1124831_22.stderr parsl.dataflow.dflow INFO: Standard output for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/6d9b71bd-b72a-4334-aff2-8035b4b1d808_characterizeImage_1124831_22.stdout parsl.dataflow.dflow INFO: Standard error for task 571 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/6d9b71bd-b72a-4334-aff2-8035b4b1d808_characterizeImage_1124831_22.stderr parsl.dataflow.dflow INFO: Task 124 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bf0cf53df923392f159bd4cd82bf4ba with result from task 124 parsl.dataflow.dflow DEBUG: Task 124 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 125 has memoization hash 0ce72d97fbbd3480112ce501d4f47fb8 parsl.dataflow.memoization INFO: Task 125 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 125 try 0 launched on executor multi with executor id 666 parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/7611d62a-1aff-4f2d-b994-f0f47abbf966_calibrate_1124831_28.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/7611d62a-1aff-4f2d-b994-f0f47abbf966_calibrate_1124831_28.stderr parsl.dataflow.dflow INFO: Standard output for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/ccebdbbb-3d11-404b-862f-619e2428bc10_characterizeImage_1124831_28.stdout parsl.dataflow.dflow INFO: Standard error for task 124 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/ccebdbbb-3d11-404b-862f-619e2428bc10_characterizeImage_1124831_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:13 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 397 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc0d34c19bbbd7f34c6d01299dd8bab8 with result from task 397 parsl.dataflow.dflow DEBUG: Task 397 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 398 has memoization hash c3781f310ec439a94b6c783bac56999f parsl.dataflow.memoization INFO: Task 398 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 398 try 0 launched on executor multi with executor id 667 parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/d5447731-dd06-4e1b-8227-d8fa5a6286e5_calibrate_1124828_46.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/d5447731-dd06-4e1b-8227-d8fa5a6286e5_calibrate_1124828_46.stderr parsl.dataflow.dflow INFO: Standard output for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/15bb7c4f-475f-4548-a0d0-4bcc0a2e34b3_characterizeImage_1124828_46.stdout parsl.dataflow.dflow INFO: Standard error for task 397 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/15bb7c4f-475f-4548-a0d0-4bcc0a2e34b3_characterizeImage_1124828_46.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 7 try 1 failed parsl.dataflow.dflow ERROR: Task 7 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 88040c6e7c7cf0d31ccdaf37bb808a6e with result from task 7 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 8 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 8 try 0 failed parsl.dataflow.dflow INFO: Task 8 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ebf66df3-03b5-4f71-a067-79db3ccd808f_calibrate_1124704_2.stdout parsl.dataflow.dflow INFO: Standard error for task 8 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ebf66df3-03b5-4f71-a067-79db3ccd808f_calibrate_1124704_2.stderr parsl.dataflow.dflow INFO: Standard output for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bdc00ba4-6611-4e7e-9998-eb57597fa382_characterizeImage_1124704_2.stdout parsl.dataflow.dflow INFO: Standard error for task 7 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bdc00ba4-6611-4e7e-9998-eb57597fa382_characterizeImage_1124704_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:14 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 523 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1f91212833f56cad9632761f4e1d3d8 with result from task 523 parsl.dataflow.dflow DEBUG: Task 523 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 524 has memoization hash 103880317c3968ae686ad04ba44885b0 parsl.dataflow.memoization INFO: Task 524 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 524 try 0 launched on executor multi with executor id 668 parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/645d0830-5720-48c8-b9b2-df68b8df241a_calibrate_1124831_51.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/645d0830-5720-48c8-b9b2-df68b8df241a_calibrate_1124831_51.stderr parsl.dataflow.dflow INFO: Standard output for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/cd284b62-d70c-4c13-9451-3f60634717ba_characterizeImage_1124831_51.stdout parsl.dataflow.dflow INFO: Standard error for task 523 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/cd284b62-d70c-4c13-9451-3f60634717ba_characterizeImage_1124831_51.stderr parsl.dataflow.dflow INFO: Task 25 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 74ebe4b61b2f700b751b5080a618dcc8 with result from task 25 parsl.dataflow.dflow DEBUG: Task 25 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 26 has memoization hash c92466e146db939895edbef77e14f34f parsl.dataflow.memoization INFO: Task 26 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 26 try 0 launched on executor multi with executor id 669 parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/cde1c650-0607-4ed6-a84b-14071e7ba660_calibrate_1124828_51.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/cde1c650-0607-4ed6-a84b-14071e7ba660_calibrate_1124828_51.stderr parsl.dataflow.dflow INFO: Standard output for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/385951e3-a59a-4a84-8e1e-d99124dc93c8_characterizeImage_1124828_51.stdout parsl.dataflow.dflow INFO: Standard error for task 25 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/385951e3-a59a-4a84-8e1e-d99124dc93c8_characterizeImage_1124828_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:15 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 577 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c0169a60f4292fb6dfe4e09defac0a8 with result from task 577 parsl.dataflow.dflow DEBUG: Task 577 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 578 has memoization hash 76c51aebd97b102067a1f06916e2b7e5 parsl.dataflow.memoization INFO: Task 578 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969ef20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 578 try 0 launched on executor multi with executor id 670 parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/63ed195d-e62b-4530-976b-ef9470da7b03_calibrate_1124831_55.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/63ed195d-e62b-4530-976b-ef9470da7b03_calibrate_1124831_55.stderr parsl.dataflow.dflow INFO: Standard output for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/d9d76f9e-a9d6-4ce7-8868-57f4d6169e38_characterizeImage_1124831_55.stdout parsl.dataflow.dflow INFO: Standard error for task 577 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/d9d76f9e-a9d6-4ce7-8868-57f4d6169e38_characterizeImage_1124831_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:16 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 154 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 17729a766d58041840975c4e52baef54 with result from task 154 parsl.dataflow.dflow DEBUG: Task 154 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 155 has memoization hash 3fec8cf7665b15f340cff042e4d1c4ba parsl.dataflow.memoization INFO: Task 155 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8f100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 155 try 0 launched on executor multi with executor id 671 parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/21d3ec24-6717-4923-894a-ad03e174b095_calibrate_1124831_59.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/21d3ec24-6717-4923-894a-ad03e174b095_calibrate_1124831_59.stderr parsl.dataflow.dflow INFO: Standard output for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/3f0cc665-28bd-44a8-b1b5-b1d2c21152bf_characterizeImage_1124831_59.stdout parsl.dataflow.dflow INFO: Standard error for task 154 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/3f0cc665-28bd-44a8-b1b5-b1d2c21152bf_characterizeImage_1124831_59.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:18 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:19 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 184 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6baa4d79cb28f464de4e28ff7f8f6d4 with result from task 184 parsl.dataflow.dflow DEBUG: Task 184 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 185 has memoization hash 048ccb333097adaf1ff32d454173ebf3 parsl.dataflow.memoization INFO: Task 185 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adad080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 185 try 0 launched on executor multi with executor id 672 parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/4ee3d11a-9927-42d8-96fe-aa13d17dc53f_calibrate_1124828_41.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/4ee3d11a-9927-42d8-96fe-aa13d17dc53f_calibrate_1124828_41.stderr parsl.dataflow.dflow INFO: Standard output for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/13e8112e-c442-4968-8b3a-19d18f8531c0_characterizeImage_1124828_41.stdout parsl.dataflow.dflow INFO: Standard error for task 184 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/13e8112e-c442-4968-8b3a-19d18f8531c0_characterizeImage_1124828_41.stderr parsl.dataflow.dflow INFO: Task 334 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8036c21b5e2f339721ae3882a2b6b864 with result from task 334 parsl.dataflow.dflow DEBUG: Task 334 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 335 has memoization hash 09101877d7d3e8d8ef8d26d50c8c9bd0 parsl.dataflow.memoization INFO: Task 335 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ced40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 335 try 0 launched on executor multi with executor id 673 parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/cf367206-74fe-407f-8812-19ba9f6f38d7_calibrate_1124831_43.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/cf367206-74fe-407f-8812-19ba9f6f38d7_calibrate_1124831_43.stderr parsl.dataflow.dflow INFO: Standard output for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/e47b1e95-cbae-4893-919b-2ade846dcd70_characterizeImage_1124831_43.stdout parsl.dataflow.dflow INFO: Standard error for task 334 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/e47b1e95-cbae-4893-919b-2ade846dcd70_characterizeImage_1124831_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:19 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 391 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f43ab3802ee4349672de393db6beaab0 with result from task 391 parsl.dataflow.dflow DEBUG: Task 391 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 392 has memoization hash 83b64723061f95dba494bf25fe254bc8 parsl.dataflow.memoization INFO: Task 392 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 392 try 0 launched on executor multi with executor id 674 parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/48e16807-15c0-4d46-ae9b-a54336786d5d_calibrate_1124828_30.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/48e16807-15c0-4d46-ae9b-a54336786d5d_calibrate_1124828_30.stderr parsl.dataflow.dflow INFO: Standard output for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/c56d0a4d-d0ef-4586-8db1-d774ce530d85_characterizeImage_1124828_30.stdout parsl.dataflow.dflow INFO: Standard error for task 391 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/c56d0a4d-d0ef-4586-8db1-d774ce530d85_characterizeImage_1124828_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:20 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 493 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc79aa1cea4d9fef7617c42a8bde89e8 with result from task 493 parsl.dataflow.dflow DEBUG: Task 493 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 494 has memoization hash 162d0c20ce23573ef27af89b582af006 parsl.dataflow.memoization INFO: Task 494 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529850cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 494 try 0 launched on executor multi with executor id 675 parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/99ae0455-9089-4e02-8ee3-3f6ae1a88997_calibrate_1124828_20.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/99ae0455-9089-4e02-8ee3-3f6ae1a88997_calibrate_1124828_20.stderr parsl.dataflow.dflow INFO: Standard output for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0caafbae-1fa3-4c1e-8169-750ed23ea692_characterizeImage_1124828_20.stdout parsl.dataflow.dflow INFO: Standard error for task 493 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0caafbae-1fa3-4c1e-8169-750ed23ea692_characterizeImage_1124828_20.stderr parsl.dataflow.dflow INFO: Task 433 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 807aafe0f8994d790c578ef441f19432 with result from task 433 parsl.dataflow.dflow DEBUG: Task 433 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 434 has memoization hash 1a08ee0edf32928167d71f3464fe4e6e parsl.dataflow.memoization INFO: Task 434 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 434 try 0 launched on executor multi with executor id 676 parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/ddea7783-92f1-47bf-8e7e-98872f29e5e1_calibrate_1124828_36.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/ddea7783-92f1-47bf-8e7e-98872f29e5e1_calibrate_1124828_36.stderr parsl.dataflow.dflow INFO: Standard output for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/d5610802-5adf-429b-bf0a-f6f6632a1f16_characterizeImage_1124828_36.stdout parsl.dataflow.dflow INFO: Standard error for task 433 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/d5610802-5adf-429b-bf0a-f6f6632a1f16_characterizeImage_1124828_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:22 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 529 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24058e00161baaad6ae478514cbab109 with result from task 529 parsl.dataflow.dflow DEBUG: Task 529 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 530 has memoization hash dbf8f14396eaa375a9473bfcaca51bdf parsl.dataflow.memoization INFO: Task 530 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298539c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 530 try 0 launched on executor multi with executor id 677 parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a9adc866-6156-4a91-a82c-c1f83ae0316b_calibrate_1124828_50.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a9adc866-6156-4a91-a82c-c1f83ae0316b_calibrate_1124828_50.stderr parsl.dataflow.dflow INFO: Standard output for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/9eb8a533-5fdd-48a4-82f8-18e43206a0cc_characterizeImage_1124828_50.stdout parsl.dataflow.dflow INFO: Standard error for task 529 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/9eb8a533-5fdd-48a4-82f8-18e43206a0cc_characterizeImage_1124828_50.stderr parsl.dataflow.dflow INFO: Task 631 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b9cbfdce314c62a097e72b3663b5c96 with result from task 631 parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:23 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow DEBUG: Task 631 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 632 has memoization hash 56609facf1705f845baf42ba34567468 parsl.dataflow.memoization INFO: Task 632 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ea160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 632 try 0 launched on executor multi with executor id 678 parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/29acc1eb-5e11-4835-b7ec-b2e1b61fd7f8_calibrate_1124828_14.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/29acc1eb-5e11-4835-b7ec-b2e1b61fd7f8_calibrate_1124828_14.stderr parsl.dataflow.dflow INFO: Standard output for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/76ab926f-8fa6-4a79-9ca1-1a878448a2b4_characterizeImage_1124828_14.stdout parsl.dataflow.dflow INFO: Standard error for task 631 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/76ab926f-8fa6-4a79-9ca1-1a878448a2b4_characterizeImage_1124828_14.stderr parsl.dataflow.dflow INFO: Task 73 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb6316345b7bd63a9aaa6f96c770ae4d with result from task 73 parsl.dataflow.dflow DEBUG: Task 73 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 74 has memoization hash b8b4c1a80916d57112d59d09af1dcd4c parsl.dataflow.memoization INFO: Task 74 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad65c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 74 try 0 launched on executor multi with executor id 679 parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/c7079ebb-db94-415d-93ee-17fb488ee03c_calibrate_1124831_58.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/c7079ebb-db94-415d-93ee-17fb488ee03c_calibrate_1124831_58.stderr parsl.dataflow.dflow INFO: Standard output for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/c19f084a-edb7-4d4d-8670-7654e3e70ec7_characterizeImage_1124831_58.stdout parsl.dataflow.dflow INFO: Standard error for task 73 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/c19f084a-edb7-4d4d-8670-7654e3e70ec7_characterizeImage_1124831_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:24 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 628 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2e93ea032a900d2a454692dbb06f1a1e with result from task 628 parsl.dataflow.dflow DEBUG: Task 628 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 629 has memoization hash 8d434c693f1600aced80ba264f337f59 parsl.dataflow.memoization INFO: Task 629 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ea480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 629 try 0 launched on executor multi with executor id 680 parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/93d1524c-2d13-45e0-94d9-075be0d5c558_calibrate_1124831_35.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/93d1524c-2d13-45e0-94d9-075be0d5c558_calibrate_1124831_35.stderr parsl.dataflow.dflow INFO: Standard output for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/3f198998-89e0-4665-9e19-64f339f80f51_characterizeImage_1124831_35.stdout parsl.dataflow.dflow INFO: Standard error for task 628 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/3f198998-89e0-4665-9e19-64f339f80f51_characterizeImage_1124831_35.stderr parsl.dataflow.dflow INFO: Task 265 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e83fc8634710256ff90dcc5f7ac9a72 with result from task 265 parsl.dataflow.dflow DEBUG: Task 265 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 266 has memoization hash ebe73d59e55599ce357174813aa0f4c8 parsl.dataflow.memoization INFO: Task 266 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aa520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 266 try 0 launched on executor multi with executor id 681 parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/2c6ccf6d-8ce1-4f7b-81f0-6292471b2bdc_calibrate_1124831_18.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/2c6ccf6d-8ce1-4f7b-81f0-6292471b2bdc_calibrate_1124831_18.stderr parsl.dataflow.dflow INFO: Standard output for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/7228e3be-d514-48e8-b0d0-0e250a381e58_characterizeImage_1124831_18.stdout parsl.dataflow.dflow INFO: Standard error for task 265 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/7228e3be-d514-48e8-b0d0-0e250a381e58_characterizeImage_1124831_18.stderr parsl.dataflow.dflow INFO: Task 187 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42855f71ca9dddedb3c0e4386a2865e4 with result from task 187 parsl.dataflow.dflow DEBUG: Task 187 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 188 has memoization hash 5e07233474a6068e1cdeb596c35c8566 parsl.dataflow.memoization INFO: Task 188 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adad3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 188 try 0 launched on executor multi with executor id 682 parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/5173662b-31b4-4b73-bcf5-34f2943d5b60_calibrate_1124828_42.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/5173662b-31b4-4b73-bcf5-34f2943d5b60_calibrate_1124828_42.stderr parsl.dataflow.dflow INFO: Standard output for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a19296e8-3699-4a35-8cad-14c87a7b38c9_characterizeImage_1124828_42.stdout parsl.dataflow.dflow INFO: Standard error for task 187 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a19296e8-3699-4a35-8cad-14c87a7b38c9_characterizeImage_1124828_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:25 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 445 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3e7ff70d6f268359b387d70484e2a17 with result from task 445 parsl.dataflow.dflow DEBUG: Task 445 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 446 has memoization hash 6a65a1cc8d1c6e0131308f001aea5290 parsl.dataflow.memoization INFO: Task 446 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981e520> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 446 try 0 launched on executor multi with executor id 683 parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/1f83ff55-7c0f-4f6f-b7ab-2b24604863df_calibrate_1124828_45.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/1f83ff55-7c0f-4f6f-b7ab-2b24604863df_calibrate_1124828_45.stderr parsl.dataflow.dflow INFO: Standard output for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0c471b40-2636-4ba2-8b0e-ead64edef852_characterizeImage_1124828_45.stdout parsl.dataflow.dflow INFO: Standard error for task 445 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0c471b40-2636-4ba2-8b0e-ead64edef852_characterizeImage_1124828_45.stderr parsl.dataflow.dflow INFO: Task 751 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7649e6606d6f9a14a3d401bf8b780b25 with result from task 751 parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:26 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow DEBUG: Task 751 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 752 has memoization hash 11656a062f8ade835d92f43fc67cee15 parsl.dataflow.memoization INFO: Task 752 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 752 try 0 launched on executor multi with executor id 684 parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a2c6538f-badd-4370-9599-f325a65c3818_calibrate_1124831_11.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a2c6538f-badd-4370-9599-f325a65c3818_calibrate_1124831_11.stderr parsl.dataflow.dflow INFO: Standard output for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5b5b3f2c-06e3-4190-9ea9-5504df1e2393_characterizeImage_1124831_11.stdout parsl.dataflow.dflow INFO: Standard error for task 751 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/5b5b3f2c-06e3-4190-9ea9-5504df1e2393_characterizeImage_1124831_11.stderr parsl.dataflow.dflow INFO: Task 406 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc53327fe930ac7f26660c2f3c1e7cb2 with result from task 406 parsl.dataflow.dflow DEBUG: Task 406 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 407 has memoization hash 5f3438cfd9d9d9885707b7b3f085ef65 parsl.dataflow.memoization INFO: Task 407 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eb240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 407 try 0 launched on executor multi with executor id 685 parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/62b1a9b4-29e9-4066-aa3a-c968e27f2813_calibrate_1124828_37.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/62b1a9b4-29e9-4066-aa3a-c968e27f2813_calibrate_1124828_37.stderr parsl.dataflow.dflow INFO: Standard output for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0b0d8ad1-3a1e-4966-86ca-de170b060f01_characterizeImage_1124828_37.stdout parsl.dataflow.dflow INFO: Standard error for task 406 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0b0d8ad1-3a1e-4966-86ca-de170b060f01_characterizeImage_1124828_37.stderr parsl.dataflow.dflow INFO: Task 22 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5db88f2d7071a90b93304a0cd6cb7917 with result from task 22 parsl.dataflow.dflow DEBUG: Task 22 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 23 has memoization hash bcde225ec57bd5f3865a97b82032df08 parsl.dataflow.memoization INFO: Task 23 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 23 try 0 launched on executor multi with executor id 686 parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f5c4a657-79f1-4503-abd4-51205663bb7b_calibrate_1124828_21.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f5c4a657-79f1-4503-abd4-51205663bb7b_calibrate_1124828_21.stderr parsl.dataflow.dflow INFO: Standard output for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/71cda3a8-370b-44d3-82b5-ac5d628495e5_characterizeImage_1124828_21.stdout parsl.dataflow.dflow INFO: Standard error for task 22 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/71cda3a8-370b-44d3-82b5-ac5d628495e5_characterizeImage_1124828_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:27 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 727 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fd48c0d2f4a6ef63d8790447327958f with result from task 727 parsl.dataflow.dflow DEBUG: Task 727 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 728 has memoization hash 69ce951539f7dd14c4d55575c23f89ba parsl.dataflow.memoization INFO: Task 728 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 728 try 0 launched on executor multi with executor id 687 parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/3e728ba3-30a7-4000-ab4f-49ed2c3e4b5f_calibrate_1124828_49.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/3e728ba3-30a7-4000-ab4f-49ed2c3e4b5f_calibrate_1124828_49.stderr parsl.dataflow.dflow INFO: Standard output for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/20715782-354a-4bcc-bd53-0ce506a63acd_characterizeImage_1124828_49.stdout parsl.dataflow.dflow INFO: Standard error for task 727 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/20715782-354a-4bcc-bd53-0ce506a63acd_characterizeImage_1124828_49.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:28 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 103 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a828042b0e86f29918bc0dfe27d2614b with result from task 103 parsl.dataflow.dflow DEBUG: Task 103 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 104 has memoization hash d8f67af7fc96a66e4152d7da7fefc1c8 parsl.dataflow.memoization INFO: Task 104 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67ba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 104 try 0 launched on executor multi with executor id 688 parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/c6a16d25-4e7c-44ff-8b6b-8a849503a3e7_calibrate_1124710_22.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/c6a16d25-4e7c-44ff-8b6b-8a849503a3e7_calibrate_1124710_22.stderr parsl.dataflow.dflow INFO: Standard output for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e974e6db-23cd-45cb-84a7-b4908f1a7850_characterizeImage_1124710_22.stdout parsl.dataflow.dflow INFO: Standard error for task 103 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e974e6db-23cd-45cb-84a7-b4908f1a7850_characterizeImage_1124710_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:29 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 676 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e8962a5e87707f40e60e7f6df2b9c70 with result from task 676 parsl.dataflow.dflow DEBUG: Task 676 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 677 has memoization hash 15389f9f4b1c4f77628210e38f5ab6e9 parsl.dataflow.memoization INFO: Task 677 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529739300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 677 try 0 launched on executor multi with executor id 689 parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/ee617dd9-b8e5-4929-9d4d-9a74e69301c9_calibrate_1124831_10.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/ee617dd9-b8e5-4929-9d4d-9a74e69301c9_calibrate_1124831_10.stderr parsl.dataflow.dflow INFO: Standard output for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/abcd9725-7e72-4b25-8d67-67809a297596_characterizeImage_1124831_10.stdout parsl.dataflow.dflow INFO: Standard error for task 676 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/abcd9725-7e72-4b25-8d67-67809a297596_characterizeImage_1124831_10.stderr parsl.dataflow.dflow INFO: Task 691 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98313d3abf97c60c7779df7664af418c with result from task 691 parsl.dataflow.dflow DEBUG: Task 691 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 692 has memoization hash 5451db0da0292acd02401f200987ea74 parsl.dataflow.memoization INFO: Task 692 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973a340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 692 try 0 launched on executor multi with executor id 690 parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/d78867cc-bea0-43c8-84da-8840712ba6e3_calibrate_1124828_53.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/d78867cc-bea0-43c8-84da-8840712ba6e3_calibrate_1124828_53.stderr parsl.dataflow.dflow INFO: Standard output for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/494232c3-befd-423c-a436-084ed2d7c04f_characterizeImage_1124828_53.stdout parsl.dataflow.dflow INFO: Standard error for task 691 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/494232c3-befd-423c-a436-084ed2d7c04f_characterizeImage_1124828_53.stderr parsl.dataflow.dflow INFO: Task 91 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd1588f1f40fa911a40ef34d99f83235 with result from task 91 parsl.dataflow.dflow DEBUG: Task 91 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 92 has memoization hash 93f1d6d780912aedb08575c426ab908a parsl.dataflow.memoization INFO: Task 92 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad66f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 92 try 0 launched on executor multi with executor id 691 parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/49c4b003-32aa-4ce8-9605-10d69731c5e9_calibrate_1124831_25.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/49c4b003-32aa-4ce8-9605-10d69731c5e9_calibrate_1124831_25.stderr parsl.dataflow.dflow INFO: Standard output for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/fdcb3497-b34a-40a4-b2f4-bae18717b6f6_characterizeImage_1124831_25.stdout parsl.dataflow.dflow INFO: Standard error for task 91 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/fdcb3497-b34a-40a4-b2f4-bae18717b6f6_characterizeImage_1124831_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:30 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:31 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 619 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e3c3c9b60f7b888ffc01b82534b5a6bf with result from task 619 parsl.dataflow.dflow DEBUG: Task 619 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 620 has memoization hash 9730da99ee13aee5afd2aeb7475936a7 parsl.dataflow.memoization INFO: Task 620 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e94e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 620 try 0 launched on executor multi with executor id 692 parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/cefaf84b-d60e-4ca0-8c17-4ff606a213b2_calibrate_1124828_55.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/cefaf84b-d60e-4ca0-8c17-4ff606a213b2_calibrate_1124828_55.stderr parsl.dataflow.dflow INFO: Standard output for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/6c917eda-39fa-42ee-af5c-58c6a69d9ce2_characterizeImage_1124828_55.stdout parsl.dataflow.dflow INFO: Standard error for task 619 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/6c917eda-39fa-42ee-af5c-58c6a69d9ce2_characterizeImage_1124828_55.stderr parsl.dataflow.dflow INFO: Task 736 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 419e98f5e12ac6a05e8a80d35b48c3a1 with result from task 736 parsl.dataflow.dflow DEBUG: Task 736 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 737 has memoization hash 78b3844274f3f79d338d0c6398995b61 parsl.dataflow.memoization INFO: Task 737 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 737 try 0 launched on executor multi with executor id 693 parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/b75673da-fc6e-4c40-a5e3-428dc2d74bc0_calibrate_1124828_38.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/b75673da-fc6e-4c40-a5e3-428dc2d74bc0_calibrate_1124828_38.stderr parsl.dataflow.dflow INFO: Standard output for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/3dd8ac04-7e6e-411d-8018-f69513f330ab_characterizeImage_1124828_38.stdout parsl.dataflow.dflow INFO: Standard error for task 736 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/3dd8ac04-7e6e-411d-8018-f69513f330ab_characterizeImage_1124828_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:32 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 385 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2bb8a9d39c7bf654190ca2c499f46fc with result from task 385 parsl.dataflow.dflow DEBUG: Task 385 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 386 has memoization hash ccf1d1757c6939b683377b21d2a5a5ff parsl.dataflow.memoization INFO: Task 386 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 386 try 0 launched on executor multi with executor id 694 parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/5f6480c5-35fd-494b-af05-0febbcb48064_calibrate_1124828_48.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/5f6480c5-35fd-494b-af05-0febbcb48064_calibrate_1124828_48.stderr parsl.dataflow.dflow INFO: Standard output for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/95303cea-2320-4c24-b7c0-9c5c3235eaaf_characterizeImage_1124828_48.stdout parsl.dataflow.dflow INFO: Standard error for task 385 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/95303cea-2320-4c24-b7c0-9c5c3235eaaf_characterizeImage_1124828_48.stderr parsl.dataflow.dflow INFO: Task 361 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7d80cfd623503c087721e2562c035e84 with result from task 361 parsl.dataflow.dflow DEBUG: Task 361 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 362 has memoization hash a4ed41033651c86dcc9ae6e8a5c66ae3 parsl.dataflow.memoization INFO: Task 362 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e89a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 362 try 0 launched on executor multi with executor id 695 parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d2cbb77c-2982-4bac-bdd8-ce663118ca1e_calibrate_1124707_8.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d2cbb77c-2982-4bac-bdd8-ce663118ca1e_calibrate_1124707_8.stderr parsl.dataflow.dflow INFO: Standard output for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/f8439488-1eea-451b-bab2-652c930c284c_characterizeImage_1124707_8.stdout parsl.dataflow.dflow INFO: Standard error for task 361 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/f8439488-1eea-451b-bab2-652c930c284c_characterizeImage_1124707_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:33 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 565 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9bf1d1a042876a0a7ff8ee6fe885af02 with result from task 565 parsl.dataflow.dflow DEBUG: Task 565 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 566 has memoization hash a2c0069405c97118d2d7a972cd23c9d7 parsl.dataflow.memoization INFO: Task 566 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969dda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 566 try 0 launched on executor multi with executor id 696 parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/3dc83a89-f9d5-4637-b270-4bfda7253cd2_calibrate_1124828_33.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/3dc83a89-f9d5-4637-b270-4bfda7253cd2_calibrate_1124828_33.stderr parsl.dataflow.dflow INFO: Standard output for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/8b9386b3-1f02-4349-b97f-fc88ead6eaae_characterizeImage_1124828_33.stdout parsl.dataflow.dflow INFO: Standard error for task 565 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/8b9386b3-1f02-4349-b97f-fc88ead6eaae_characterizeImage_1124828_33.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:34 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:35 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 442 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 73615d31b8ad90cfad990143713d51fd with result from task 442 parsl.dataflow.dflow DEBUG: Task 442 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 443 has memoization hash d96c659a4cd5614bc30c354d8b294506 parsl.dataflow.memoization INFO: Task 443 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981e200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 443 try 0 launched on executor multi with executor id 697 parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/0c906e6c-3a9e-49a2-9042-a96552ed3d60_calibrate_1124704_10.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/0c906e6c-3a9e-49a2-9042-a96552ed3d60_calibrate_1124704_10.stderr parsl.dataflow.dflow INFO: Standard output for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d44a6066-7a9b-4d81-95ac-65f9359a67bd_characterizeImage_1124704_10.stdout parsl.dataflow.dflow INFO: Standard error for task 442 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d44a6066-7a9b-4d81-95ac-65f9359a67bd_characterizeImage_1124704_10.stderr parsl.dataflow.dflow INFO: Task 139 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 27d987c86d803bf0263664fe14a20c39 with result from task 139 parsl.dataflow.dflow DEBUG: Task 139 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 140 has memoization hash 48e9018084362d0302b86781de6d4262 parsl.dataflow.memoization INFO: Task 140 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8e160> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 140 try 0 launched on executor multi with executor id 698 parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/0022f63b-6b87-402b-a837-4d40977f77fc_calibrate_1124710_33.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/0022f63b-6b87-402b-a837-4d40977f77fc_calibrate_1124710_33.stderr parsl.dataflow.dflow INFO: Standard output for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1321d47b-a29d-4aad-bbca-8a26d71dacec_characterizeImage_1124710_33.stdout parsl.dataflow.dflow INFO: Standard error for task 139 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1321d47b-a29d-4aad-bbca-8a26d71dacec_characterizeImage_1124710_33.stderr parsl.dataflow.dflow INFO: Task 694 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eaeaad8f9f54e8d376c57a760158c850 with result from task 694 parsl.dataflow.dflow DEBUG: Task 694 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 695 has memoization hash 0712b93e757c9dd292afa885c2d9c98b parsl.dataflow.memoization INFO: Task 695 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973a3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 695 try 0 launched on executor multi with executor id 699 parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/d745d679-ff4c-4160-ab35-2b2425e7654c_calibrate_1124828_25.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/d745d679-ff4c-4160-ab35-2b2425e7654c_calibrate_1124828_25.stderr parsl.dataflow.dflow INFO: Standard output for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0976be9d-81ed-4361-91df-87563f5a3d04_characterizeImage_1124828_25.stdout parsl.dataflow.dflow INFO: Standard error for task 694 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0976be9d-81ed-4361-91df-87563f5a3d04_characterizeImage_1124828_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:38 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 298 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98162062fc27142b01ad1e2c3efcd361 with result from task 298 parsl.dataflow.dflow DEBUG: Task 298 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 299 has memoization hash 42f8628fa8fe6659db0243e7cdd49829 parsl.dataflow.memoization INFO: Task 299 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cc7c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 299 try 0 launched on executor multi with executor id 700 parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8a9c78e9-7bde-4af8-a76e-9be36c7d6b65_calibrate_1124828_43.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8a9c78e9-7bde-4af8-a76e-9be36c7d6b65_calibrate_1124828_43.stderr parsl.dataflow.dflow INFO: Standard output for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0e4487b8-7d83-4560-9fa9-62cf50006c09_characterizeImage_1124828_43.stdout parsl.dataflow.dflow INFO: Standard error for task 298 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/0e4487b8-7d83-4560-9fa9-62cf50006c09_characterizeImage_1124828_43.stderr parsl.dataflow.dflow INFO: Task 538 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 395a9a4394dade268936568c3e172384 with result from task 538 parsl.dataflow.dflow DEBUG: Task 538 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash e171923504aa2c4ccbb268b2f6830fc6 parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 0 launched on executor multi with executor id 701 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/83c52c40-c27e-433c-98bf-f968d4880528_calibrate_1124707_31.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/83c52c40-c27e-433c-98bf-f968d4880528_calibrate_1124707_31.stderr parsl.dataflow.dflow INFO: Standard output for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3c89eaff-75e8-4229-84f2-a14aec595db7_characterizeImage_1124707_31.stdout parsl.dataflow.dflow INFO: Standard error for task 538 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3c89eaff-75e8-4229-84f2-a14aec595db7_characterizeImage_1124707_31.stderr parsl.dataflow.dflow INFO: Task 646 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7eb81ff02f7cebc92f820e74601210a with result from task 646 parsl.dataflow.dflow DEBUG: Task 646 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 647 has memoization hash 46c4c7a341a10785dc67b17d4acb2eba parsl.dataflow.memoization INFO: Task 647 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eb740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 647 try 0 launched on executor multi with executor id 702 parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/91deba8e-51bb-44fc-b21f-d91b75d9876a_calibrate_1124710_1.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/91deba8e-51bb-44fc-b21f-d91b75d9876a_calibrate_1124710_1.stderr parsl.dataflow.dflow INFO: Standard output for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7ce38cdd-896b-41f4-8edc-e7baf1da2008_characterizeImage_1124710_1.stdout parsl.dataflow.dflow INFO: Standard error for task 646 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7ce38cdd-896b-41f4-8edc-e7baf1da2008_characterizeImage_1124710_1.stderr parsl.dataflow.dflow INFO: Task 367 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 06b29b672eb8dcffb38d2f26955ac174 with result from task 367 parsl.dataflow.dflow DEBUG: Task 367 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 368 has memoization hash 9ab802d36b4911f7d1b76bc0e296f0ec parsl.dataflow.memoization INFO: Task 368 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 368 try 0 launched on executor multi with executor id 703 parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/317d451a-f2be-460c-8ce8-e24167358622_calibrate_1124710_26.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/317d451a-f2be-460c-8ce8-e24167358622_calibrate_1124710_26.stderr parsl.dataflow.dflow INFO: Standard output for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/850eb6e1-4341-4588-aabd-e9cbd5d24ce7_characterizeImage_1124710_26.stdout parsl.dataflow.dflow INFO: Standard error for task 367 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/850eb6e1-4341-4588-aabd-e9cbd5d24ce7_characterizeImage_1124710_26.stderr parsl.dataflow.dflow INFO: Task 346 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b5ad90fa8deb4a235b18420c8e40fa5 with result from task 346 parsl.dataflow.dflow DEBUG: Task 346 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 347 has memoization hash 92b130887d8fd02a7b03736459d98572 parsl.dataflow.memoization INFO: Task 347 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cf9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 347 try 0 launched on executor multi with executor id 704 parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ed2c1157-8947-4245-8162-18de1cb44ba6_calibrate_1124710_27.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ed2c1157-8947-4245-8162-18de1cb44ba6_calibrate_1124710_27.stderr parsl.dataflow.dflow INFO: Standard output for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/181bb79e-c255-406a-afd7-b30869848aa9_characterizeImage_1124710_27.stdout parsl.dataflow.dflow INFO: Standard error for task 346 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/181bb79e-c255-406a-afd7-b30869848aa9_characterizeImage_1124710_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:40 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 514 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0f166ed8f11feb9a4410ea66e505709 with result from task 514 parsl.dataflow.dflow DEBUG: Task 514 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 515 has memoization hash 785f9a788ca8ecd6b0e9852958c0c8e6 parsl.dataflow.memoization INFO: Task 515 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 515 try 0 launched on executor multi with executor id 705 parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/a8fbb37a-55a6-46c6-9f25-07166dd5ba4b_calibrate_1124710_62.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/a8fbb37a-55a6-46c6-9f25-07166dd5ba4b_calibrate_1124710_62.stderr parsl.dataflow.dflow INFO: Standard output for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f4852dfc-b0ee-465e-beed-7f80c02ecc57_characterizeImage_1124710_62.stdout parsl.dataflow.dflow INFO: Standard error for task 514 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f4852dfc-b0ee-465e-beed-7f80c02ecc57_characterizeImage_1124710_62.stderr parsl.dataflow.dflow INFO: Task 730 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54742a04aaaacdff68ed1fd97dc7c950 with result from task 730 parsl.dataflow.dflow DEBUG: Task 730 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 731 has memoization hash 88111540117c35374158c7bcc476f857 parsl.dataflow.memoization INFO: Task 731 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977fec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 731 try 0 launched on executor multi with executor id 706 parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/6bc76c9e-1f3f-4ad8-9697-70b83104d397_calibrate_1124831_62.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/6bc76c9e-1f3f-4ad8-9697-70b83104d397_calibrate_1124831_62.stderr parsl.dataflow.dflow INFO: Standard output for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/2d62d27b-6ea3-4f0f-92d6-c73c201d2d5f_characterizeImage_1124831_62.stdout parsl.dataflow.dflow INFO: Standard error for task 730 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/2d62d27b-6ea3-4f0f-92d6-c73c201d2d5f_characterizeImage_1124831_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:41 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 118 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 60ce5713370ace0630595eda7cc0a70e with result from task 118 parsl.dataflow.dflow DEBUG: Task 118 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 119 has memoization hash d430aabf67fae363b834c9542b0bdbcd parsl.dataflow.memoization INFO: Task 119 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8cb80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 119 try 0 launched on executor multi with executor id 707 parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7dd962a7-d977-470e-b57f-16a6913e45ec_calibrate_1124710_9.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7dd962a7-d977-470e-b57f-16a6913e45ec_calibrate_1124710_9.stderr parsl.dataflow.dflow INFO: Standard output for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b73ca140-b3f4-4d52-a554-cfe8d2da317b_characterizeImage_1124710_9.stdout parsl.dataflow.dflow INFO: Standard error for task 118 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b73ca140-b3f4-4d52-a554-cfe8d2da317b_characterizeImage_1124710_9.stderr parsl.dataflow.dflow INFO: Task 142 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2d36859990c41b78afb02ec636f60d77 with result from task 142 parsl.dataflow.dflow DEBUG: Task 142 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 143 has memoization hash c5f36ac44f44e3dde3d3169cd86201e2 parsl.dataflow.memoization INFO: Task 143 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8e480> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 143 try 0 launched on executor multi with executor id 708 parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1e902908-2c2f-4de9-ac26-827a81507929_calibrate_1124710_32.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1e902908-2c2f-4de9-ac26-827a81507929_calibrate_1124710_32.stderr parsl.dataflow.dflow INFO: Standard output for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d6dfe81a-9aa9-4fd9-84a4-e1a0e7ba49e0_characterizeImage_1124710_32.stdout parsl.dataflow.dflow INFO: Standard error for task 142 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d6dfe81a-9aa9-4fd9-84a4-e1a0e7ba49e0_characterizeImage_1124710_32.stderr parsl.dataflow.dflow INFO: Task 136 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e4195b767d06833ea1d06fea6806fce with result from task 136 parsl.dataflow.dflow DEBUG: Task 136 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 137 has memoization hash 6d0cc177a685388813585f83a8931438 parsl.dataflow.memoization INFO: Task 137 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8de40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 137 try 0 launched on executor multi with executor id 709 parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d126533f-e0fd-4015-9bd1-579db1c41347_calibrate_1124707_36.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d126533f-e0fd-4015-9bd1-579db1c41347_calibrate_1124707_36.stderr parsl.dataflow.dflow INFO: Standard output for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/930935fe-1e32-4372-ad85-2febbfddf6eb_characterizeImage_1124707_36.stdout parsl.dataflow.dflow INFO: Standard error for task 136 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/930935fe-1e32-4372-ad85-2febbfddf6eb_characterizeImage_1124707_36.stderr parsl.dataflow.dflow INFO: Task 430 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 071dc5911b0a5e18935e448ffa6d35a9 with result from task 430 parsl.dataflow.dflow DEBUG: Task 430 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 431 has memoization hash 8aa3d0e025e0f80a395a03c88b1b4e23 parsl.dataflow.memoization INFO: Task 431 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981d1c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 431 try 0 launched on executor multi with executor id 710 parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e9551fbe-b12d-480e-b3c0-79e129ef664f_calibrate_1124707_42.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e9551fbe-b12d-480e-b3c0-79e129ef664f_calibrate_1124707_42.stderr parsl.dataflow.dflow INFO: Standard output for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/7b9795e0-d915-4ed7-aef8-aab9fae36e94_characterizeImage_1124707_42.stdout parsl.dataflow.dflow INFO: Standard error for task 430 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/7b9795e0-d915-4ed7-aef8-aab9fae36e94_characterizeImage_1124707_42.stderr parsl.dataflow.dflow INFO: Task 400 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eeabde7777b8da66a82d16b9c1dd3c9d with result from task 400 parsl.dataflow.dflow DEBUG: Task 400 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 401 has memoization hash a205c45a778fa89a1b38f7aaed08e54a parsl.dataflow.memoization INFO: Task 401 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ea980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 401 try 0 launched on executor multi with executor id 711 parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/14b34b14-041a-4ae6-b4c9-4ef320975c8b_calibrate_1124704_46.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/14b34b14-041a-4ae6-b4c9-4ef320975c8b_calibrate_1124704_46.stderr parsl.dataflow.dflow INFO: Standard output for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d438c735-ed49-47fa-a595-a7dd50f442e8_characterizeImage_1124704_46.stdout parsl.dataflow.dflow INFO: Standard error for task 400 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d438c735-ed49-47fa-a595-a7dd50f442e8_characterizeImage_1124704_46.stderr parsl.dataflow.dflow INFO: Task 130 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 71dfd99e663a26bd4d06bf74de83b693 with result from task 130 parsl.dataflow.dflow DEBUG: Task 130 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 131 has memoization hash 8be55351e218efcc620404de725d5550 parsl.dataflow.memoization INFO: Task 131 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 131 try 0 launched on executor multi with executor id 712 parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/9a84967e-3ccc-44cb-9af2-e7725d673076_calibrate_1124704_22.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/9a84967e-3ccc-44cb-9af2-e7725d673076_calibrate_1124704_22.stderr parsl.dataflow.dflow INFO: Standard output for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/35394d1c-cfec-4420-b599-0cc935a7a44b_characterizeImage_1124704_22.stdout parsl.dataflow.dflow INFO: Standard error for task 130 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/35394d1c-cfec-4420-b599-0cc935a7a44b_characterizeImage_1124704_22.stderr parsl.dataflow.dflow INFO: Task 802 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 374c1b87d7cae629d8ffafbcd1921b60 with result from task 802 parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:41 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow DEBUG: Task 802 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 803 has memoization hash 62a816b66796c91209939bcd50c11707 parsl.dataflow.memoization INFO: Task 803 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cbe20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 803 try 0 launched on executor multi with executor id 713 parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3aabd228-c636-4d20-8b37-9021c6e4df8d_calibrate_1124831_41.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3aabd228-c636-4d20-8b37-9021c6e4df8d_calibrate_1124831_41.stderr parsl.dataflow.dflow INFO: Standard output for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/573f5f93-aa7f-4135-b6be-7847371f0a26_characterizeImage_1124831_41.stdout parsl.dataflow.dflow INFO: Standard error for task 802 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/573f5f93-aa7f-4135-b6be-7847371f0a26_characterizeImage_1124831_41.stderr parsl.dataflow.dflow INFO: Task 649 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 65c739d038cf4db0aa2a0d499a6b7d68 with result from task 649 parsl.dataflow.dflow DEBUG: Task 649 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 650 has memoization hash b9748bbca32b585bbdbe7dd58ca09e20 parsl.dataflow.memoization INFO: Task 650 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 650 try 0 launched on executor multi with executor id 714 parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a2220a61-1200-4532-8a83-d6260d199684_calibrate_1124831_7.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a2220a61-1200-4532-8a83-d6260d199684_calibrate_1124831_7.stderr parsl.dataflow.dflow INFO: Standard output for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/dd274776-ab8b-4651-acc9-286dae18a29b_characterizeImage_1124831_7.stdout parsl.dataflow.dflow INFO: Standard error for task 649 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/dd274776-ab8b-4651-acc9-286dae18a29b_characterizeImage_1124831_7.stderr parsl.dataflow.dflow INFO: Task 541 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29d8a1722629c00c0e38bb4a88b26b97 with result from task 541 parsl.dataflow.dflow DEBUG: Task 541 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 542 has memoization hash d81d1380ae0e7392a0408f36c45bcb65 parsl.dataflow.memoization INFO: Task 542 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969c0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 542 try 0 launched on executor multi with executor id 715 parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/0b01d29a-fcf3-4366-ba93-53d2418305db_calibrate_1124704_33.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/0b01d29a-fcf3-4366-ba93-53d2418305db_calibrate_1124704_33.stderr parsl.dataflow.dflow INFO: Standard output for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/77026af8-5bad-4205-9629-e4ff84033216_characterizeImage_1124704_33.stdout parsl.dataflow.dflow INFO: Standard error for task 541 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/77026af8-5bad-4205-9629-e4ff84033216_characterizeImage_1124704_33.stderr parsl.dataflow.dflow INFO: Task 97 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6722f02da540abe148d2fc5e6bcc4757 with result from task 97 parsl.dataflow.dflow DEBUG: Task 97 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 98 has memoization hash d16d8ed1a671348e14508aa75e52b52b parsl.dataflow.memoization INFO: Task 98 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 98 try 0 launched on executor multi with executor id 716 parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/94f6edee-d6f1-4008-87f0-8d3fe1e34e40_calibrate_1124704_14.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/94f6edee-d6f1-4008-87f0-8d3fe1e34e40_calibrate_1124704_14.stderr parsl.dataflow.dflow INFO: Standard output for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/2b15b24b-ca99-4e51-a7b1-65e97ade9f6c_characterizeImage_1124704_14.stdout parsl.dataflow.dflow INFO: Standard error for task 97 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/2b15b24b-ca99-4e51-a7b1-65e97ade9f6c_characterizeImage_1124704_14.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 733 try 1 failed parsl.dataflow.dflow ERROR: Task 733 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 0e6631dc1017587635539c370f059183 with result from task 733 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 734 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 734 try 0 failed parsl.dataflow.dflow INFO: Task 734 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/c857240d-6ce3-45df-837e-58710da2a0fc_calibrate_1124710_2.stdout parsl.dataflow.dflow INFO: Standard error for task 734 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/c857240d-6ce3-45df-837e-58710da2a0fc_calibrate_1124710_2.stderr parsl.dataflow.dflow INFO: Standard output for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/23f01ed6-d153-43be-8984-62e5f76efe73_characterizeImage_1124710_2.stdout parsl.dataflow.dflow INFO: Standard error for task 733 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/23f01ed6-d153-43be-8984-62e5f76efe73_characterizeImage_1124710_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:42 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 412 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 87e350ccbc95679372874289b05ca2bb with result from task 412 parsl.dataflow.dflow DEBUG: Task 412 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 413 has memoization hash 3915334b5be25fe7a63ffd5a392f9e7d parsl.dataflow.memoization INFO: Task 413 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eb9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 413 try 0 launched on executor multi with executor id 717 parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cd208141-736b-430c-9b9d-62d04f85de97_calibrate_1124707_5.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cd208141-736b-430c-9b9d-62d04f85de97_calibrate_1124707_5.stderr parsl.dataflow.dflow INFO: Standard output for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e3229bb8-b47c-4bf6-b459-76ae366746e0_characterizeImage_1124707_5.stdout parsl.dataflow.dflow INFO: Standard error for task 412 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e3229bb8-b47c-4bf6-b459-76ae366746e0_characterizeImage_1124707_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:43 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 376 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e286d664317dab90959e5fe3ac8d80b1 with result from task 376 parsl.dataflow.dflow DEBUG: Task 376 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 377 has memoization hash cda33b9ebd6965336cdedb4e03e9321e parsl.dataflow.memoization INFO: Task 377 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e9940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 377 try 0 launched on executor multi with executor id 718 parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/00d1d206-0092-461c-9548-5bd629b5b056_calibrate_1124704_26.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/00d1d206-0092-461c-9548-5bd629b5b056_calibrate_1124704_26.stderr parsl.dataflow.dflow INFO: Standard output for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b51f50b9-0407-4622-bec2-874c0813e35e_characterizeImage_1124704_26.stdout parsl.dataflow.dflow INFO: Standard error for task 376 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b51f50b9-0407-4622-bec2-874c0813e35e_characterizeImage_1124704_26.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 592 try 1 failed parsl.dataflow.dflow ERROR: Task 592 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry fecb4c03f58e9e88baba6e363c229ef3 with result from task 592 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 593 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 593 try 0 failed parsl.dataflow.dflow INFO: Task 593 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ca9a0c1b-9a3d-409a-8f0b-59f912b982f8_calibrate_1124707_2.stdout parsl.dataflow.dflow INFO: Standard error for task 593 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ca9a0c1b-9a3d-409a-8f0b-59f912b982f8_calibrate_1124707_2.stderr parsl.dataflow.dflow INFO: Standard output for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3464ff69-e292-4267-b1de-41e6fa27387c_characterizeImage_1124707_2.stdout parsl.dataflow.dflow INFO: Standard error for task 592 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3464ff69-e292-4267-b1de-41e6fa27387c_characterizeImage_1124707_2.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:44 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 808 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 062f990c1a55d1953966a4614d5661a8 with result from task 808 parsl.dataflow.dflow DEBUG: Task 808 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 809 has memoization hash c6eacf41c434061aaca70ddd1f2b796a parsl.dataflow.memoization INFO: Task 809 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 809 try 0 launched on executor multi with executor id 719 parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a75da45d-62c3-49a4-be30-c4568d89da20_calibrate_1124828_57.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a75da45d-62c3-49a4-be30-c4568d89da20_calibrate_1124828_57.stderr parsl.dataflow.dflow INFO: Standard output for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/44c96ce0-af4b-457d-a9d1-7e95db4baa64_characterizeImage_1124828_57.stdout parsl.dataflow.dflow INFO: Standard error for task 808 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/44c96ce0-af4b-457d-a9d1-7e95db4baa64_characterizeImage_1124828_57.stderr parsl.dataflow.dflow INFO: Task 844 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afe2622ec39bae43ddfb03c5e724cc28 with result from task 844 parsl.dataflow.dflow DEBUG: Task 844 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 845 has memoization hash 80084c22b7b5e55f73eee99a269b4769 parsl.dataflow.memoization INFO: Task 845 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 845 try 0 launched on executor multi with executor id 720 parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1807ab7d-7594-4aee-9699-7395b6917229_calibrate_1124707_62.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1807ab7d-7594-4aee-9699-7395b6917229_calibrate_1124707_62.stderr parsl.dataflow.dflow INFO: Standard output for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/68c3cbf4-38fa-4953-8554-c5a8cae02a7f_characterizeImage_1124707_62.stdout parsl.dataflow.dflow INFO: Standard error for task 844 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/68c3cbf4-38fa-4953-8554-c5a8cae02a7f_characterizeImage_1124707_62.stderr parsl.dataflow.dflow INFO: Task 670 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd9cf50cc26323bd100a8a858396b430 with result from task 670 parsl.dataflow.dflow DEBUG: Task 670 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 671 has memoization hash 83119de6c2148debaf9fed92ac4ae03b parsl.dataflow.memoization INFO: Task 671 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529738d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 671 try 0 launched on executor multi with executor id 721 parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/2f96bf99-8c50-4cc3-baed-81d254d7aa41_calibrate_1124828_47.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/2f96bf99-8c50-4cc3-baed-81d254d7aa41_calibrate_1124828_47.stderr parsl.dataflow.dflow INFO: Standard output for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/c0167ffd-2eb8-4087-a2ce-c415120cc0d1_characterizeImage_1124828_47.stdout parsl.dataflow.dflow INFO: Standard error for task 670 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/c0167ffd-2eb8-4087-a2ce-c415120cc0d1_characterizeImage_1124828_47.stderr parsl.dataflow.dflow INFO: Task 145 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d1328491c5fdddfdf0274eb1a2db2976 with result from task 145 parsl.dataflow.dflow DEBUG: Task 145 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 146 has memoization hash 5b1c4c29b2d431b80dabce693aef8d30 parsl.dataflow.memoization INFO: Task 146 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8e7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 146 try 0 launched on executor multi with executor id 722 parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/8e1623ab-99ca-4f7f-8869-48c3efe8c16d_calibrate_1124710_23.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/8e1623ab-99ca-4f7f-8869-48c3efe8c16d_calibrate_1124710_23.stderr parsl.dataflow.dflow INFO: Standard output for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/306514a6-39ec-4d8a-81ec-e48a1a5bb822_characterizeImage_1124710_23.stdout parsl.dataflow.dflow INFO: Standard error for task 145 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/306514a6-39ec-4d8a-81ec-e48a1a5bb822_characterizeImage_1124710_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:46 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 580 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e741c3ebe598a30d4db4b3ff0849814d with result from task 580 parsl.dataflow.dflow DEBUG: Task 580 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 581 has memoization hash 84306f90dca26c5ccd1b417cd0d08ed5 parsl.dataflow.memoization INFO: Task 581 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969f060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 581 try 0 launched on executor multi with executor id 723 parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/74e5ecb2-076a-4834-8176-f92112501c09_calibrate_1124707_32.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/74e5ecb2-076a-4834-8176-f92112501c09_calibrate_1124707_32.stderr parsl.dataflow.dflow INFO: Standard output for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/55201bb7-0888-4988-97f2-a12092ccd529_characterizeImage_1124707_32.stdout parsl.dataflow.dflow INFO: Standard error for task 580 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/55201bb7-0888-4988-97f2-a12092ccd529_characterizeImage_1124707_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:47 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 382 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5b688a838dd4142419c4a91b242bfe1 with result from task 382 parsl.dataflow.dflow DEBUG: Task 382 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 383 has memoization hash 744c9c3c7e889843fb6a2f870eb60962 parsl.dataflow.memoization INFO: Task 383 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 383 try 0 launched on executor multi with executor id 724 parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/a81eb202-93ef-462d-9721-06f80500079b_calibrate_1124710_51.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/a81eb202-93ef-462d-9721-06f80500079b_calibrate_1124710_51.stderr parsl.dataflow.dflow INFO: Standard output for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/69ff441e-0144-40a3-b07c-2b686bfd27f1_characterizeImage_1124710_51.stdout parsl.dataflow.dflow INFO: Standard error for task 382 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/69ff441e-0144-40a3-b07c-2b686bfd27f1_characterizeImage_1124710_51.stderr parsl.dataflow.dflow INFO: Task 58 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93b1f3b25209fc2e18cb624c3b4c3bd8 with result from task 58 parsl.dataflow.dflow DEBUG: Task 58 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 59 has memoization hash 0bda1c67ed2a36a7df4b0f22e0ef0ee6 parsl.dataflow.memoization INFO: Task 59 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 59 try 0 launched on executor multi with executor id 725 parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/a4f50922-e252-4473-b5c4-d269fa1a35d0_calibrate_1124710_30.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/a4f50922-e252-4473-b5c4-d269fa1a35d0_calibrate_1124710_30.stderr parsl.dataflow.dflow INFO: Standard output for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/ebc83df7-cfa2-459c-b54d-83579493687c_characterizeImage_1124710_30.stdout parsl.dataflow.dflow INFO: Standard error for task 58 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/ebc83df7-cfa2-459c-b54d-83579493687c_characterizeImage_1124710_30.stderr parsl.dataflow.dflow INFO: Task 829 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be50399ae7b14e6bbbf16a73d1c64f2d with result from task 829 parsl.dataflow.dflow DEBUG: Task 829 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 830 has memoization hash f7ae4ffb25128aba3b5de5f6d51ffcbd parsl.dataflow.memoization INFO: Task 830 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 830 try 0 launched on executor multi with executor id 726 parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8cb8878b-db49-45a0-85fe-6fc718390e40_calibrate_1124828_56.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8cb8878b-db49-45a0-85fe-6fc718390e40_calibrate_1124828_56.stderr parsl.dataflow.dflow INFO: Standard output for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/634ccfbd-7010-464e-828f-88a4a78c5556_characterizeImage_1124828_56.stdout parsl.dataflow.dflow INFO: Standard error for task 829 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/634ccfbd-7010-464e-828f-88a4a78c5556_characterizeImage_1124828_56.stderr parsl.dataflow.dflow INFO: Task 637 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9fdf81ce2f13ad13ba1d7fb2e6daf23 with result from task 637 parsl.dataflow.dflow DEBUG: Task 637 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 638 has memoization hash 584b6e426300c975662d27b6b6b47d8a parsl.dataflow.memoization INFO: Task 638 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 638 try 0 launched on executor multi with executor id 727 parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d5d89f0d-2576-431c-9add-995eb3f58800_calibrate_1124707_19.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d5d89f0d-2576-431c-9add-995eb3f58800_calibrate_1124707_19.stderr parsl.dataflow.dflow INFO: Standard output for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/93f141fd-3d59-4a8c-be13-47ac066c22ca_characterizeImage_1124707_19.stdout parsl.dataflow.dflow INFO: Standard error for task 637 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/93f141fd-3d59-4a8c-be13-47ac066c22ca_characterizeImage_1124707_19.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:48 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 811 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 674d444084cf2deafe08759f85420d30 with result from task 811 parsl.dataflow.dflow DEBUG: Task 811 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 812 has memoization hash e4bae2d6e143a8155c822987bd110194 parsl.dataflow.memoization INFO: Task 812 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c9b20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 812 try 0 launched on executor multi with executor id 728 parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/79e4cf3f-dfdb-4eed-8a62-e6e9ef6d70ee_calibrate_1124828_6.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/79e4cf3f-dfdb-4eed-8a62-e6e9ef6d70ee_calibrate_1124828_6.stderr parsl.dataflow.dflow INFO: Standard output for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/d3a6abeb-3c46-495b-9537-a22f431bed05_characterizeImage_1124828_6.stdout parsl.dataflow.dflow INFO: Standard error for task 811 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/d3a6abeb-3c46-495b-9537-a22f431bed05_characterizeImage_1124828_6.stderr parsl.dataflow.dflow INFO: Task 106 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b840248ee0b7b5637a8cec99dc11ad75 with result from task 106 parsl.dataflow.dflow DEBUG: Task 106 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash bcd408b2aee6cf23dfa118d4f35ab8cd parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 0 launched on executor multi with executor id 729 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/526ba917-d599-4483-8704-f932cf8e8631_calibrate_1124710_31.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/526ba917-d599-4483-8704-f932cf8e8631_calibrate_1124710_31.stderr parsl.dataflow.dflow INFO: Standard output for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f78d7bdb-06e2-4e84-ae3d-94a7ca1e5927_characterizeImage_1124710_31.stdout parsl.dataflow.dflow INFO: Standard error for task 106 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f78d7bdb-06e2-4e84-ae3d-94a7ca1e5927_characterizeImage_1124710_31.stderr parsl.dataflow.dflow INFO: Task 121 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a6a216f5b6b8957c897d9ec3c22ec7a4 with result from task 121 parsl.dataflow.dflow DEBUG: Task 121 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 122 has memoization hash 6ae21e0981a4240ef8d3e1110d03177f parsl.dataflow.memoization INFO: Task 122 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 122 try 0 launched on executor multi with executor id 730 parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/65580c76-b944-40cd-9dbe-aa894d5ab01a_calibrate_1124704_50.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/65580c76-b944-40cd-9dbe-aa894d5ab01a_calibrate_1124704_50.stderr parsl.dataflow.dflow INFO: Standard output for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/27ed34c8-680f-4a4d-b10c-cabebd0c9c6f_characterizeImage_1124704_50.stdout parsl.dataflow.dflow INFO: Standard error for task 121 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/27ed34c8-680f-4a4d-b10c-cabebd0c9c6f_characterizeImage_1124704_50.stderr parsl.dataflow.dflow INFO: Task 850 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 183ac088bf5d4c1100eee0361fea52d8 with result from task 850 parsl.dataflow.dflow DEBUG: Task 850 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 851 has memoization hash 059ed7e15fd665eee53c902d5affdab2 parsl.dataflow.memoization INFO: Task 851 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 851 try 0 launched on executor multi with executor id 731 parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/da30db92-b6d9-465c-9cb5-f94498025fb5_calibrate_1124828_8.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/da30db92-b6d9-465c-9cb5-f94498025fb5_calibrate_1124828_8.stderr parsl.dataflow.dflow INFO: Standard output for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/766b7761-173f-4ef2-982b-b439c89e3180_characterizeImage_1124828_8.stdout parsl.dataflow.dflow INFO: Standard error for task 850 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/766b7761-173f-4ef2-982b-b439c89e3180_characterizeImage_1124828_8.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:49 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 763 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7165f85bcc8b47683e89b02d92a95fa7 with result from task 763 parsl.dataflow.dflow DEBUG: Task 763 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 764 has memoization hash c86d90c1d496cc724a123ba967aa3fc0 parsl.dataflow.memoization INFO: Task 764 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977ea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 764 try 0 launched on executor multi with executor id 732 parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a964d103-2452-4dd7-b35c-5bcab14e478c_calibrate_1124828_40.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a964d103-2452-4dd7-b35c-5bcab14e478c_calibrate_1124828_40.stderr parsl.dataflow.dflow INFO: Standard output for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/62bb184b-a588-43f6-a9d9-bc63c94c64a2_characterizeImage_1124828_40.stdout parsl.dataflow.dflow INFO: Standard error for task 763 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/62bb184b-a588-43f6-a9d9-bc63c94c64a2_characterizeImage_1124828_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:50 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 532 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6bc324dd48ae0246c725307fe8c315c7 with result from task 532 parsl.dataflow.dflow DEBUG: Task 532 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 533 has memoization hash f4033a874525512e131f3102f4e8791d parsl.dataflow.memoization INFO: Task 533 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529853b00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 533 try 0 launched on executor multi with executor id 733 parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/fec92767-c40f-4bcf-bbc1-ec6010be1b72_calibrate_1124710_35.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/fec92767-c40f-4bcf-bbc1-ec6010be1b72_calibrate_1124710_35.stderr parsl.dataflow.dflow INFO: Standard output for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/42a53e32-44da-498c-95a9-ac2dd1c4f9a3_characterizeImage_1124710_35.stdout parsl.dataflow.dflow INFO: Standard error for task 532 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/42a53e32-44da-498c-95a9-ac2dd1c4f9a3_characterizeImage_1124710_35.stderr parsl.dataflow.dflow INFO: Task 562 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee47230098b5aaa31b8cac8033663955 with result from task 562 parsl.dataflow.dflow DEBUG: Task 562 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 563 has memoization hash 68f36f9a52c57246cde62a25d68cf93a parsl.dataflow.memoization INFO: Task 563 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969e0c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 563 try 0 launched on executor multi with executor id 734 parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/96c86f86-aeb2-488d-9514-9528331ad979_calibrate_1124707_26.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/96c86f86-aeb2-488d-9514-9528331ad979_calibrate_1124707_26.stderr parsl.dataflow.dflow INFO: Standard output for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/b691a32b-f8da-4962-aa3d-c36928ab8193_characterizeImage_1124707_26.stdout parsl.dataflow.dflow INFO: Standard error for task 562 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/b691a32b-f8da-4962-aa3d-c36928ab8193_characterizeImage_1124707_26.stderr parsl.dataflow.dflow INFO: Task 781 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cecf837fd0cb9720206f21ac4545e0f2 with result from task 781 parsl.dataflow.dflow DEBUG: Task 781 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 782 has memoization hash c37594f51dcfbbfdc3d22ae2c4c31297 parsl.dataflow.memoization INFO: Task 782 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c84a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 782 try 0 launched on executor multi with executor id 735 parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/06e75b7a-ed34-4740-8d47-457583d1f484_calibrate_1124704_52.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/06e75b7a-ed34-4740-8d47-457583d1f484_calibrate_1124704_52.stderr parsl.dataflow.dflow INFO: Standard output for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/46ac0cf9-d797-4f86-9543-d4970f62c1fc_characterizeImage_1124704_52.stdout parsl.dataflow.dflow INFO: Standard error for task 781 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/46ac0cf9-d797-4f86-9543-d4970f62c1fc_characterizeImage_1124704_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:51 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 640 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3cefb392aeaaf224f0c375273071647a with result from task 640 parsl.dataflow.dflow DEBUG: Task 640 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 641 has memoization hash 457a009ba6af073990f6fb6894042cc0 parsl.dataflow.memoization INFO: Task 641 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 641 try 0 launched on executor multi with executor id 736 parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/35607b2a-b901-4a12-9e3c-b0bb67ee8a18_calibrate_1124707_56.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/35607b2a-b901-4a12-9e3c-b0bb67ee8a18_calibrate_1124707_56.stderr parsl.dataflow.dflow INFO: Standard output for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/23b1db91-d656-4f10-893a-3a46427618c8_characterizeImage_1124707_56.stdout parsl.dataflow.dflow INFO: Standard error for task 640 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/23b1db91-d656-4f10-893a-3a46427618c8_characterizeImage_1124707_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:52 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 280 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e28ee521b9b21e1f237d4164fd1baf69 with result from task 280 parsl.dataflow.dflow DEBUG: Task 280 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 281 has memoization hash e7e26d30eee56f747004ee4c85b9f4d0 parsl.dataflow.memoization INFO: Task 281 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ab4c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 281 try 0 launched on executor multi with executor id 737 parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4ab71671-00e2-4a47-82cd-fe15ff7cb093_calibrate_1124831_52.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4ab71671-00e2-4a47-82cd-fe15ff7cb093_calibrate_1124831_52.stderr parsl.dataflow.dflow INFO: Standard output for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/435de374-89bb-45a6-84c6-c95557499d03_characterizeImage_1124831_52.stdout parsl.dataflow.dflow INFO: Standard error for task 280 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/435de374-89bb-45a6-84c6-c95557499d03_characterizeImage_1124831_52.stderr parsl.dataflow.dflow INFO: Task 739 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 604b3101a249b740bf140b2833239e16 with result from task 739 parsl.dataflow.dflow DEBUG: Task 739 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 740 has memoization hash cfca881c40258ad115791adf7ef76d50 parsl.dataflow.memoization INFO: Task 740 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977cea0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 740 try 0 launched on executor multi with executor id 738 parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/77f774a1-76c4-470c-9944-906f4e2f1d28_calibrate_1124707_35.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/77f774a1-76c4-470c-9944-906f4e2f1d28_calibrate_1124707_35.stderr parsl.dataflow.dflow INFO: Standard output for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/7a9dafea-43bb-4cf1-baaa-691b79839bdf_characterizeImage_1124707_35.stdout parsl.dataflow.dflow INFO: Standard error for task 739 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/7a9dafea-43bb-4cf1-baaa-691b79839bdf_characterizeImage_1124707_35.stderr parsl.dataflow.dflow INFO: Task 499 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb1f86e07a582ad63fc3a1f2081cd70a with result from task 499 parsl.dataflow.dflow DEBUG: Task 499 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 500 has memoization hash aee7c85a6b9736e255774f553986d9c3 parsl.dataflow.memoization INFO: Task 500 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298518a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 500 try 0 launched on executor multi with executor id 739 parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4f20c361-b818-428b-ba02-26f82a5d742d_calibrate_1124831_13.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4f20c361-b818-428b-ba02-26f82a5d742d_calibrate_1124831_13.stderr parsl.dataflow.dflow INFO: Standard output for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f8f9f41a-95e2-4e7e-a477-3084d55627fa_characterizeImage_1124831_13.stdout parsl.dataflow.dflow INFO: Standard error for task 499 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f8f9f41a-95e2-4e7e-a477-3084d55627fa_characterizeImage_1124831_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:53 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 772 try 1 failed parsl.dataflow.dflow ERROR: Task 772 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 3de12400f6b8a0bb3edf4a850a7177da with result from task 772 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 773 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 773 try 0 failed parsl.dataflow.dflow INFO: Task 773 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/57d384b6-258d-46e3-9c8a-b2bf8d3610c1_calibrate_1124831_2.stdout parsl.dataflow.dflow INFO: Standard error for task 773 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/57d384b6-258d-46e3-9c8a-b2bf8d3610c1_calibrate_1124831_2.stderr parsl.dataflow.dflow INFO: Standard output for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/142f6ce8-f81b-476d-ac52-5acf2bf1c6d5_characterizeImage_1124831_2.stdout parsl.dataflow.dflow INFO: Standard error for task 772 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/142f6ce8-f81b-476d-ac52-5acf2bf1c6d5_characterizeImage_1124831_2.stderr parsl.dataflow.dflow INFO: Task 625 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee3176055ded0617d4eae17e04e35fa7 with result from task 625 parsl.dataflow.dflow DEBUG: Task 625 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 626 has memoization hash b858bed7d8a8e170d6b6db338d261a12 parsl.dataflow.memoization INFO: Task 626 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ea7a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 626 try 0 launched on executor multi with executor id 740 parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/8103d01c-dd47-4fb0-b816-1aebdf2f2d05_calibrate_1124710_45.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/8103d01c-dd47-4fb0-b816-1aebdf2f2d05_calibrate_1124710_45.stderr parsl.dataflow.dflow INFO: Standard output for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/47aa9eb7-f525-484c-9193-1ed9647e81c5_characterizeImage_1124710_45.stdout parsl.dataflow.dflow INFO: Standard error for task 625 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/47aa9eb7-f525-484c-9193-1ed9647e81c5_characterizeImage_1124710_45.stderr parsl.dataflow.dflow INFO: Task 316 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 251eb9ec6c3e81df29fd7b5976195f14 with result from task 316 parsl.dataflow.dflow DEBUG: Task 316 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 317 has memoization hash e190e36c70e6fb5d83b687d5456b2e0a parsl.dataflow.memoization INFO: Task 317 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cda80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 317 try 0 launched on executor multi with executor id 741 parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4d2fb781-ade2-4b80-b3c4-7f6e36953563_calibrate_1124831_57.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4d2fb781-ade2-4b80-b3c4-7f6e36953563_calibrate_1124831_57.stderr parsl.dataflow.dflow INFO: Standard output for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/9f37f68f-152e-42df-bf37-448918cbd08a_characterizeImage_1124831_57.stdout parsl.dataflow.dflow INFO: Standard error for task 316 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/9f37f68f-152e-42df-bf37-448918cbd08a_characterizeImage_1124831_57.stderr parsl.dataflow.dflow INFO: Task 817 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 854c64df2d6cba2203c9fb93069fd54a with result from task 817 parsl.dataflow.dflow DEBUG: Task 817 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 818 has memoization hash afdd9b0cd317dea1760bd8ad8d26f0d1 parsl.dataflow.memoization INFO: Task 818 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295ca020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 818 try 0 launched on executor multi with executor id 742 parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/8c7811af-2c78-4e71-a9a2-844b093f5a93_calibrate_1124831_17.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/8c7811af-2c78-4e71-a9a2-844b093f5a93_calibrate_1124831_17.stderr parsl.dataflow.dflow INFO: Standard output for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/84cf8a55-adc4-4c1b-8289-3a2c77c4d254_characterizeImage_1124831_17.stdout parsl.dataflow.dflow INFO: Standard error for task 817 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/84cf8a55-adc4-4c1b-8289-3a2c77c4d254_characterizeImage_1124831_17.stderr parsl.dataflow.dflow INFO: Task 682 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e45239e11a4b98f59bd21aa3373385de with result from task 682 parsl.dataflow.dflow DEBUG: Task 682 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 683 has memoization hash 065d532909913e592ab9a41d8398e052 parsl.dataflow.memoization INFO: Task 683 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297399e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 683 try 0 launched on executor multi with executor id 743 parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1393cf79-ae33-40c7-8273-da7f81407f67_calibrate_1124710_11.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1393cf79-ae33-40c7-8273-da7f81407f67_calibrate_1124710_11.stderr parsl.dataflow.dflow INFO: Standard output for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/aff787f5-9c21-4cd7-84a6-e85110b98875_characterizeImage_1124710_11.stdout parsl.dataflow.dflow INFO: Standard error for task 682 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/aff787f5-9c21-4cd7-84a6-e85110b98875_characterizeImage_1124710_11.stderr parsl.dataflow.dflow INFO: Task 64 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 407685f006aca373a62865a4e84c737e with result from task 64 parsl.dataflow.dflow DEBUG: Task 64 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 65 has memoization hash b68a719ca8fefebcdbe6fdfeff467372 parsl.dataflow.memoization INFO: Task 65 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad65300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 65 try 0 launched on executor multi with executor id 744 parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ddaa9ef4-fa3a-4a2a-93bb-3757464ab963_calibrate_1124704_9.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ddaa9ef4-fa3a-4a2a-93bb-3757464ab963_calibrate_1124704_9.stderr parsl.dataflow.dflow INFO: Standard output for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d39f632f-7c64-4136-bc70-30ffb055370f_characterizeImage_1124704_9.stdout parsl.dataflow.dflow INFO: Standard error for task 64 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d39f632f-7c64-4136-bc70-30ffb055370f_characterizeImage_1124704_9.stderr parsl.dataflow.dflow INFO: Task 655 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ad776372aeababb4bcdb46c27c806795 with result from task 655 parsl.dataflow.dflow DEBUG: Task 655 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 656 has memoization hash 794710c5bfbea690046d1a0751fb7904 parsl.dataflow.memoization INFO: Task 656 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969f9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 656 try 0 launched on executor multi with executor id 745 parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/0bda3662-abb4-4e14-a7e2-b163739a86e0_calibrate_1124710_19.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/0bda3662-abb4-4e14-a7e2-b163739a86e0_calibrate_1124710_19.stderr parsl.dataflow.dflow INFO: Standard output for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d0c93e16-506f-4a62-9426-066a2b91cffd_characterizeImage_1124710_19.stdout parsl.dataflow.dflow INFO: Standard error for task 655 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d0c93e16-506f-4a62-9426-066a2b91cffd_characterizeImage_1124710_19.stderr parsl.dataflow.dflow INFO: Task 589 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0461687baec30768f4fca159e307850f with result from task 589 parsl.dataflow.dflow DEBUG: Task 589 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 590 has memoization hash 90c10adc6442d42db0eabc56fdb4c8fe parsl.dataflow.memoization INFO: Task 590 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 590 try 0 launched on executor multi with executor id 746 parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/347633b5-5ebf-48ea-892b-33e832aa2f88_calibrate_1124704_35.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/347633b5-5ebf-48ea-892b-33e832aa2f88_calibrate_1124704_35.stderr parsl.dataflow.dflow INFO: Standard output for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b6137e75-60d3-4244-b81c-18c00175ae5d_characterizeImage_1124704_35.stdout parsl.dataflow.dflow INFO: Standard error for task 589 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b6137e75-60d3-4244-b81c-18c00175ae5d_characterizeImage_1124704_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:54 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 274 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a07af5abc9a8614cab8f192528f216e0 with result from task 274 parsl.dataflow.dflow DEBUG: Task 274 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 275 has memoization hash a01f944200a0a54c1e7bcf1e9b15b074 parsl.dataflow.memoization INFO: Task 275 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aae80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 275 try 0 launched on executor multi with executor id 747 parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cb88bf09-d788-402d-9e36-f26b43697ff4_calibrate_1124707_3.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cb88bf09-d788-402d-9e36-f26b43697ff4_calibrate_1124707_3.stderr parsl.dataflow.dflow INFO: Standard output for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/02a2948e-15b7-4a07-a988-0ef015106b86_characterizeImage_1124707_3.stdout parsl.dataflow.dflow INFO: Standard error for task 274 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/02a2948e-15b7-4a07-a988-0ef015106b86_characterizeImage_1124707_3.stderr parsl.dataflow.dflow INFO: Task 241 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f70fea85bce9c6382623d847a747ee75 with result from task 241 parsl.dataflow.dflow DEBUG: Task 241 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 242 has memoization hash a36ca329d0cc44c6f1c625f28bffc7c3 parsl.dataflow.memoization INFO: Task 242 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 242 try 0 launched on executor multi with executor id 748 parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/9b583028-6d7a-43fa-be1b-82939ada8db0_calibrate_1124704_8.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/9b583028-6d7a-43fa-be1b-82939ada8db0_calibrate_1124704_8.stderr parsl.dataflow.dflow INFO: Standard output for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/5e752f47-83ec-4f69-aa8b-6bb7470b0da1_characterizeImage_1124704_8.stdout parsl.dataflow.dflow INFO: Standard error for task 241 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/5e752f47-83ec-4f69-aa8b-6bb7470b0da1_characterizeImage_1124704_8.stderr parsl.dataflow.dflow INFO: Task 544 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 26f7b51194ca5ba2638c446a84134ca0 with result from task 544 parsl.dataflow.dflow DEBUG: Task 544 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 545 has memoization hash 493fd56eee42e3c70663a0eb70a2dee3 parsl.dataflow.memoization INFO: Task 545 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969c400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 545 try 0 launched on executor multi with executor id 749 parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ab3f8d26-8aa8-4f73-8b7e-b897ce496856_calibrate_1124710_29.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ab3f8d26-8aa8-4f73-8b7e-b897ce496856_calibrate_1124710_29.stderr parsl.dataflow.dflow INFO: Standard output for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/51583bb4-d4e7-4552-a6cc-159a431dfaab_characterizeImage_1124710_29.stdout parsl.dataflow.dflow INFO: Standard error for task 544 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/51583bb4-d4e7-4552-a6cc-159a431dfaab_characterizeImage_1124710_29.stderr parsl.dataflow.dflow INFO: Task 223 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 718485514fca17902288277ab4e0a623 with result from task 223 parsl.dataflow.dflow DEBUG: Task 223 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 224 has memoization hash 29527bedce098781a85b2dd2346a0e74 parsl.dataflow.memoization INFO: Task 224 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaf920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 224 try 0 launched on executor multi with executor id 750 parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/b1f9002e-672d-44de-993b-98c39a333ce0_calibrate_1124704_56.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/b1f9002e-672d-44de-993b-98c39a333ce0_calibrate_1124704_56.stderr parsl.dataflow.dflow INFO: Standard output for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7962c7e0-a434-4559-9a12-4e99cb8ad35a_characterizeImage_1124704_56.stdout parsl.dataflow.dflow INFO: Standard error for task 223 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7962c7e0-a434-4559-9a12-4e99cb8ad35a_characterizeImage_1124704_56.stderr parsl.dataflow.dflow INFO: Task 295 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fcb96002a96a69971f9a931e17bd9a88 with result from task 295 parsl.dataflow.dflow DEBUG: Task 295 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 296 has memoization hash 12da49202063734aa45b018478a41172 parsl.dataflow.memoization INFO: Task 296 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cc4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 296 try 0 launched on executor multi with executor id 751 parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1b95afe3-989b-4072-9966-4a0fb0c2a4e9_calibrate_1124707_46.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1b95afe3-989b-4072-9966-4a0fb0c2a4e9_calibrate_1124707_46.stderr parsl.dataflow.dflow INFO: Standard output for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/b9d9326f-e440-4a8c-b5a3-11e87fc74a38_characterizeImage_1124707_46.stdout parsl.dataflow.dflow INFO: Standard error for task 295 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/b9d9326f-e440-4a8c-b5a3-11e87fc74a38_characterizeImage_1124707_46.stderr parsl.dataflow.dflow INFO: Task 481 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e1c374dc5eada5bed4c13be03706bfe with result from task 481 parsl.dataflow.dflow DEBUG: Task 481 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 482 has memoization hash c7bdd787bfb382408c5f5306cac818ed parsl.dataflow.memoization INFO: Task 482 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529850220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 482 try 0 launched on executor multi with executor id 752 parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/b5796c22-e0cc-4ffd-aabc-08cb413631c5_calibrate_1124831_16.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/b5796c22-e0cc-4ffd-aabc-08cb413631c5_calibrate_1124831_16.stderr parsl.dataflow.dflow INFO: Standard output for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/354f14e4-6ccf-46e7-b39c-14a4a741f111_characterizeImage_1124831_16.stdout parsl.dataflow.dflow INFO: Standard error for task 481 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/354f14e4-6ccf-46e7-b39c-14a4a741f111_characterizeImage_1124831_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:55 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 325 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 19b3f05c1d34097de6e06f0ff97f6fb2 with result from task 325 parsl.dataflow.dflow DEBUG: Task 325 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 326 has memoization hash 5491a03ba50c7f32bc06610bd4ec6ea8 parsl.dataflow.memoization INFO: Task 326 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ce3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 326 try 0 launched on executor multi with executor id 753 parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/9e1c70a0-c52b-4cb0-b9b9-e999fb82ebee_calibrate_1124710_15.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/9e1c70a0-c52b-4cb0-b9b9-e999fb82ebee_calibrate_1124710_15.stderr parsl.dataflow.dflow INFO: Standard output for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1ad38249-5484-4675-b0d7-ddca04c6d22b_characterizeImage_1124710_15.stdout parsl.dataflow.dflow INFO: Standard error for task 325 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1ad38249-5484-4675-b0d7-ddca04c6d22b_characterizeImage_1124710_15.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 370 try 1 failed parsl.dataflow.dflow ERROR: Task 370 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry f2e3ec23c7297a32110a9ffdedcd1967 with result from task 370 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 371 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 371 try 0 failed parsl.dataflow.dflow INFO: Task 371 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/86f2469d-9805-4033-9394-1144e3397d0c_calibrate_1124831_61.stdout parsl.dataflow.dflow INFO: Standard error for task 371 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/86f2469d-9805-4033-9394-1144e3397d0c_calibrate_1124831_61.stderr parsl.dataflow.dflow INFO: Standard output for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/45b82ec0-bfba-47fd-baa4-5f7cbe3dceab_characterizeImage_1124831_61.stdout parsl.dataflow.dflow INFO: Standard error for task 370 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/45b82ec0-bfba-47fd-baa4-5f7cbe3dceab_characterizeImage_1124831_61.stderr parsl.dataflow.dflow INFO: Task 598 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb3b03c0e9d0a47f383bf85fc2db66cc with result from task 598 parsl.dataflow.dflow DEBUG: Task 598 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 599 has memoization hash 3f027e8cce3fa5d534021554b9cf2918 parsl.dataflow.memoization INFO: Task 599 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eaac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 599 try 0 launched on executor multi with executor id 754 parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8f1b9710-be69-47f2-8561-e219418846ec_calibrate_1124704_1.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8f1b9710-be69-47f2-8561-e219418846ec_calibrate_1124704_1.stderr parsl.dataflow.dflow INFO: Standard output for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/e36ff1ff-ac1f-439b-960f-b10930bc622b_characterizeImage_1124704_1.stdout parsl.dataflow.dflow INFO: Standard error for task 598 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/e36ff1ff-ac1f-439b-960f-b10930bc622b_characterizeImage_1124704_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:56 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 868 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01fe69571c20250909e31a1d68101b01 with result from task 868 parsl.dataflow.dflow DEBUG: Task 868 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 869 has memoization hash 4cea1991d5fcef5cdb6016bbe38d2c67 parsl.dataflow.memoization INFO: Task 869 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529609f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 869 try 0 launched on executor multi with executor id 755 parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/0da66c8f-37a7-41a5-bf52-ca7376e61120_calibrate_1124831_26.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/0da66c8f-37a7-41a5-bf52-ca7376e61120_calibrate_1124831_26.stderr parsl.dataflow.dflow INFO: Standard output for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/97ede7ed-10e2-4315-94b3-023473b5bdae_characterizeImage_1124831_26.stdout parsl.dataflow.dflow INFO: Standard error for task 868 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/97ede7ed-10e2-4315-94b3-023473b5bdae_characterizeImage_1124831_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:57 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 256 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 621a639ffcee3a2cdc40d33e0ccd78ef with result from task 256 parsl.dataflow.dflow DEBUG: Task 256 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 257 has memoization hash c5d29ecb817ace1ef88f2e2abc13a8d3 parsl.dataflow.memoization INFO: Task 257 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 257 try 0 launched on executor multi with executor id 756 parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/624ec647-9674-4b6a-9eab-dbd66091a771_calibrate_1124707_16.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/624ec647-9674-4b6a-9eab-dbd66091a771_calibrate_1124707_16.stderr parsl.dataflow.dflow INFO: Standard output for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/582bd3cd-9676-48dc-948e-ecce28bc3dda_characterizeImage_1124707_16.stdout parsl.dataflow.dflow INFO: Standard error for task 256 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/582bd3cd-9676-48dc-948e-ecce28bc3dda_characterizeImage_1124707_16.stderr parsl.dataflow.dflow INFO: Task 568 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c37715cd66faf585f3f3aa9a5c4263e0 with result from task 568 parsl.dataflow.dflow DEBUG: Task 568 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 569 has memoization hash 0907d0a6fb29f20ea8885e53bf8561e6 parsl.dataflow.memoization INFO: Task 569 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 569 try 0 launched on executor multi with executor id 757 parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/983e0f51-f60f-4998-b8a6-a300c0a2203d_calibrate_1124831_37.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/983e0f51-f60f-4998-b8a6-a300c0a2203d_calibrate_1124831_37.stderr parsl.dataflow.dflow INFO: Standard output for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f6057aac-61be-4ce6-ab8d-4d8dee8140fa_characterizeImage_1124831_37.stdout parsl.dataflow.dflow INFO: Standard error for task 568 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f6057aac-61be-4ce6-ab8d-4d8dee8140fa_characterizeImage_1124831_37.stderr parsl.dataflow.dflow INFO: Task 271 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 483378a11ce5ff9d93913532bde57240 with result from task 271 parsl.dataflow.dflow DEBUG: Task 271 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 272 has memoization hash 34d86fff3f2c6460006b20021c8b2dcb parsl.dataflow.memoization INFO: Task 272 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 272 try 0 launched on executor multi with executor id 758 parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1bd5ad9c-532b-45cd-8039-44da2e8d5eee_calibrate_1124707_47.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1bd5ad9c-532b-45cd-8039-44da2e8d5eee_calibrate_1124707_47.stderr parsl.dataflow.dflow INFO: Standard output for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/59f04c55-fc13-48a9-8d5f-c098f1ad20e8_characterizeImage_1124707_47.stdout parsl.dataflow.dflow INFO: Standard error for task 271 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/59f04c55-fc13-48a9-8d5f-c098f1ad20e8_characterizeImage_1124707_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:58 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 133 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1163c22824518dce76827f7f5399057e with result from task 133 parsl.dataflow.dflow DEBUG: Task 133 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 134 has memoization hash c4898231263cc8ecbd7504e6a9b7e95a parsl.dataflow.memoization INFO: Task 134 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8db20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 134 try 0 launched on executor multi with executor id 759 parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/e14e416e-ed94-4a9a-a12a-653c13e18d01_calibrate_1124831_20.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/e14e416e-ed94-4a9a-a12a-653c13e18d01_calibrate_1124831_20.stderr parsl.dataflow.dflow INFO: Standard output for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/02b285b8-ec63-44a8-9f25-8b244634ee82_characterizeImage_1124831_20.stdout parsl.dataflow.dflow INFO: Standard error for task 133 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/02b285b8-ec63-44a8-9f25-8b244634ee82_characterizeImage_1124831_20.stderr parsl.dataflow.dflow INFO: Task 217 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 63aa7fcca340e6a64570dd40a3a472ba with result from task 217 parsl.dataflow.dflow DEBUG: Task 217 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 218 has memoization hash 7438f370f14044e1897a5b65ac7e93ac parsl.dataflow.memoization INFO: Task 218 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaf2e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 218 try 0 launched on executor multi with executor id 760 parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/8d6c444a-e44f-43cc-84a7-a3b11db1afc6_calibrate_1124710_10.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/8d6c444a-e44f-43cc-84a7-a3b11db1afc6_calibrate_1124710_10.stderr parsl.dataflow.dflow INFO: Standard output for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/66a0d956-60bd-4c94-b433-79ae576cf6a8_characterizeImage_1124710_10.stdout parsl.dataflow.dflow INFO: Standard error for task 217 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/66a0d956-60bd-4c94-b433-79ae576cf6a8_characterizeImage_1124710_10.stderr parsl.dataflow.dflow INFO: Task 214 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3545fc8e8cbdf3bd5b564afb475b7b4e with result from task 214 parsl.dataflow.dflow DEBUG: Task 214 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 215 has memoization hash 5fbc0fefaeaaec8c5fcbee3506242917 parsl.dataflow.memoization INFO: Task 215 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaefc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 215 try 0 launched on executor multi with executor id 761 parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/b970717d-091a-4d55-8f3a-b5690ca6ac6a_calibrate_1124704_34.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/b970717d-091a-4d55-8f3a-b5690ca6ac6a_calibrate_1124704_34.stderr parsl.dataflow.dflow INFO: Standard output for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/cd089f32-bf18-45ac-8d11-311cc2e26207_characterizeImage_1124704_34.stdout parsl.dataflow.dflow INFO: Standard error for task 214 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/cd089f32-bf18-45ac-8d11-311cc2e26207_characterizeImage_1124704_34.stderr parsl.dataflow.dflow INFO: Task 823 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 594233d685b644b2d5e2e53239ae9e5b with result from task 823 parsl.dataflow.dflow DEBUG: Task 823 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 824 has memoization hash 8c23964c1b1b7c0b0db1ac0a454b3425 parsl.dataflow.memoization INFO: Task 824 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295ca700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 824 try 0 launched on executor multi with executor id 762 parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/751bb54d-07a4-4f51-89be-9c6043833420_calibrate_1124831_12.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/751bb54d-07a4-4f51-89be-9c6043833420_calibrate_1124831_12.stderr parsl.dataflow.dflow INFO: Standard output for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/4f46a892-43a9-4ee0-86b3-7fe4cdbe25cc_characterizeImage_1124831_12.stdout parsl.dataflow.dflow INFO: Standard error for task 823 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/4f46a892-43a9-4ee0-86b3-7fe4cdbe25cc_characterizeImage_1124831_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:28:59 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 82 try 1 failed parsl.dataflow.dflow ERROR: Task 82 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app characterizeImage failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 1dbe3e95e947bbde0227f98a31162e9f with result from task 82 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Task 83 failed due to dependency failure parsl.dataflow.dflow DEBUG: Task 83 try 0 failed parsl.dataflow.dflow INFO: Task 83 failed due to dependency failure so skipping retries parsl.dataflow.memoization ERROR: Attempting to update app cache entry but hashsum is not a string key parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/22e285dc-5256-481a-b237-f84d8dae793d_calibrate_1124828_61.stdout parsl.dataflow.dflow INFO: Standard error for task 83 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/22e285dc-5256-481a-b237-f84d8dae793d_calibrate_1124828_61.stderr parsl.dataflow.dflow INFO: Standard output for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/7021b40b-e213-4f60-b138-ffdbfbf58281_characterizeImage_1124828_61.stdout parsl.dataflow.dflow INFO: Standard error for task 82 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/7021b40b-e213-4f60-b138-ffdbfbf58281_characterizeImage_1124828_61.stderr parsl.dataflow.dflow INFO: Task 52 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ea26259118583174a3ad6ba36465b6f with result from task 52 parsl.dataflow.dflow DEBUG: Task 52 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 53 has memoization hash fb0b2d8bd1cdbc8cd3626da2cb7053a5 parsl.dataflow.memoization INFO: Task 53 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad64680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 53 try 0 launched on executor multi with executor id 763 parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cc39350e-05c5-4b1e-ad7f-a2cd82e68861_calibrate_1124707_41.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cc39350e-05c5-4b1e-ad7f-a2cd82e68861_calibrate_1124707_41.stderr parsl.dataflow.dflow INFO: Standard output for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/004688bb-cf8c-4d54-a2ed-ce745ca8961e_characterizeImage_1124707_41.stdout parsl.dataflow.dflow INFO: Standard error for task 52 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/004688bb-cf8c-4d54-a2ed-ce745ca8961e_characterizeImage_1124707_41.stderr parsl.dataflow.dflow INFO: Task 526 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05672be1ecefe3919c0802c856b43af6 with result from task 526 parsl.dataflow.dflow DEBUG: Task 526 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 527 has memoization hash 7dd12fa8f62c12727f3b0ed1ab5ac876 parsl.dataflow.memoization INFO: Task 527 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 527 try 0 launched on executor multi with executor id 764 parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/668bc0fe-6487-4c53-96de-b296cb3ad825_calibrate_1124710_42.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/668bc0fe-6487-4c53-96de-b296cb3ad825_calibrate_1124710_42.stderr parsl.dataflow.dflow INFO: Standard output for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/72c3ea49-7faa-4871-a35f-f4e9b7b62d8b_characterizeImage_1124710_42.stdout parsl.dataflow.dflow INFO: Standard error for task 526 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/72c3ea49-7faa-4871-a35f-f4e9b7b62d8b_characterizeImage_1124710_42.stderr parsl.dataflow.dflow INFO: Task 622 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6aad0f2884cfc62177f90f409f90db3a with result from task 622 parsl.dataflow.dflow DEBUG: Task 622 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 623 has memoization hash 07d0676790eaa560905dda61f8d09cf1 parsl.dataflow.memoization INFO: Task 623 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e9800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 623 try 0 launched on executor multi with executor id 765 parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/a02cdab4-8cb2-40e4-b547-2720976a55f8_calibrate_1124704_29.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/a02cdab4-8cb2-40e4-b547-2720976a55f8_calibrate_1124704_29.stderr parsl.dataflow.dflow INFO: Standard output for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/62a67ab7-db44-43e4-a14e-3695b34abae4_characterizeImage_1124704_29.stdout parsl.dataflow.dflow INFO: Standard error for task 622 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/62a67ab7-db44-43e4-a14e-3695b34abae4_characterizeImage_1124704_29.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:00 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 643 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8ea1ef677d3f12b98e9764f9f3f70c0e with result from task 643 parsl.dataflow.dflow DEBUG: Task 643 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 644 has memoization hash 8f205bff390192006ba130def9d3ea46 parsl.dataflow.memoization INFO: Task 644 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eb420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 644 try 0 launched on executor multi with executor id 766 parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/768ecffc-261b-4ebc-b8cd-3cc11c642d0a_calibrate_1124707_6.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/768ecffc-261b-4ebc-b8cd-3cc11c642d0a_calibrate_1124707_6.stderr parsl.dataflow.dflow INFO: Standard output for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/b4e25147-cdae-4fa9-bfdd-626a9b1ac245_characterizeImage_1124707_6.stdout parsl.dataflow.dflow INFO: Standard error for task 643 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/b4e25147-cdae-4fa9-bfdd-626a9b1ac245_characterizeImage_1124707_6.stderr parsl.dataflow.dflow INFO: Task 310 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b20d2d4199e5bb6b2b8678cb8b6bc8f with result from task 310 parsl.dataflow.dflow DEBUG: Task 310 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 311 has memoization hash da398009589aa431749456b122c200da parsl.dataflow.memoization INFO: Task 311 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cd440> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 311 try 0 launched on executor multi with executor id 767 parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/862f4de7-835d-48ba-881a-a4e8a993c72d_calibrate_1124704_39.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/862f4de7-835d-48ba-881a-a4e8a993c72d_calibrate_1124704_39.stderr parsl.dataflow.dflow INFO: Standard output for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/0b027ecc-8728-4bd2-9ae0-b99553738615_characterizeImage_1124704_39.stdout parsl.dataflow.dflow INFO: Standard error for task 310 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/0b027ecc-8728-4bd2-9ae0-b99553738615_characterizeImage_1124704_39.stderr parsl.dataflow.dflow INFO: Task 604 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ef7b1b831a845eaeed5869092bd658f4 with result from task 604 parsl.dataflow.dflow DEBUG: Task 604 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 605 has memoization hash 23252e7a9823b8e845f92422ff7b36da parsl.dataflow.memoization INFO: Task 605 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e8540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 605 try 0 launched on executor multi with executor id 768 parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/249ef6ab-9a9a-4bae-a9ad-8e361cb51ec4_calibrate_1124710_54.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/249ef6ab-9a9a-4bae-a9ad-8e361cb51ec4_calibrate_1124710_54.stderr parsl.dataflow.dflow INFO: Standard output for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e8e98071-9bbc-48ae-8e22-ad5ae624732d_characterizeImage_1124710_54.stdout parsl.dataflow.dflow INFO: Standard error for task 604 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e8e98071-9bbc-48ae-8e22-ad5ae624732d_characterizeImage_1124710_54.stderr parsl.dataflow.dflow INFO: Task 364 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c3091b9b049bc466d187dd9a87eecc5 with result from task 364 parsl.dataflow.dflow DEBUG: Task 364 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 365 has memoization hash de4116e7bd0e5e7b33470aec12d5c89d parsl.dataflow.memoization INFO: Task 365 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8cc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 365 try 0 launched on executor multi with executor id 769 parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/f5222419-4921-4112-a25b-47e7ff3fe571_calibrate_1124704_19.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/f5222419-4921-4112-a25b-47e7ff3fe571_calibrate_1124704_19.stderr parsl.dataflow.dflow INFO: Standard output for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/9dd324a1-8db4-4a04-8db9-9fee074add49_characterizeImage_1124704_19.stdout parsl.dataflow.dflow INFO: Standard error for task 364 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/9dd324a1-8db4-4a04-8db9-9fee074add49_characterizeImage_1124704_19.stderr parsl.dataflow.dflow INFO: Task 160 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b50a4462ed1baf61d6c828f93b29c354 with result from task 160 parsl.dataflow.dflow DEBUG: Task 160 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 161 has memoization hash 08e0b42d0c8cc4491b86db46429d3eb3 parsl.dataflow.memoization INFO: Task 161 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8f740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 161 try 0 launched on executor multi with executor id 770 parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/40a352d0-8a16-4ae1-9d03-4656b985f995_calibrate_1124828_62.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/40a352d0-8a16-4ae1-9d03-4656b985f995_calibrate_1124828_62.stderr parsl.dataflow.dflow INFO: Standard output for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/61a9d9c8-e489-414a-9f82-7627149b9f8e_characterizeImage_1124828_62.stdout parsl.dataflow.dflow INFO: Standard error for task 160 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/61a9d9c8-e489-414a-9f82-7627149b9f8e_characterizeImage_1124828_62.stderr parsl.dataflow.dflow INFO: Task 193 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50d83e7c3e4648ad1364c5fedcd9fd68 with result from task 193 parsl.dataflow.dflow DEBUG: Task 193 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 194 has memoization hash 798e310dc783be490f845045fde7386a parsl.dataflow.memoization INFO: Task 194 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adad9e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 194 try 0 launched on executor multi with executor id 771 parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/96d86ae5-f619-40fe-a493-b95fb5fe8a8e_calibrate_1124828_11.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/96d86ae5-f619-40fe-a493-b95fb5fe8a8e_calibrate_1124828_11.stderr parsl.dataflow.dflow INFO: Standard output for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/31e02587-d961-4b88-9012-74a5427851af_characterizeImage_1124828_11.stdout parsl.dataflow.dflow INFO: Standard error for task 193 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/31e02587-d961-4b88-9012-74a5427851af_characterizeImage_1124828_11.stderr parsl.dataflow.dflow INFO: Task 610 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f9305c11a8e40d2e290c4aca4fc3bd24 with result from task 610 parsl.dataflow.dflow DEBUG: Task 610 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 611 has memoization hash 6daee7f891bc6d0cf37cd95ddc7625c4 parsl.dataflow.memoization INFO: Task 611 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e8a40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 611 try 0 launched on executor multi with executor id 772 parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ca919ccc-75aa-44fb-b461-fbd2aa28ef75_calibrate_1124707_18.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ca919ccc-75aa-44fb-b461-fbd2aa28ef75_calibrate_1124707_18.stderr parsl.dataflow.dflow INFO: Standard output for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/86d61e2d-f73a-459a-8135-9daafc3fb8d4_characterizeImage_1124707_18.stdout parsl.dataflow.dflow INFO: Standard error for task 610 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/86d61e2d-f73a-459a-8135-9daafc3fb8d4_characterizeImage_1124707_18.stderr parsl.dataflow.dflow INFO: Task 418 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ce646a1440c49d4ae2d1029af931870 with result from task 418 parsl.dataflow.dflow DEBUG: Task 418 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 419 has memoization hash 777fcc099c244ca8e79a5b0588037f26 parsl.dataflow.memoization INFO: Task 419 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981c040> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 419 try 0 launched on executor multi with executor id 773 parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/953cee98-635b-4b88-b0b0-c03db25c2c00_calibrate_1124710_53.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/953cee98-635b-4b88-b0b0-c03db25c2c00_calibrate_1124710_53.stderr parsl.dataflow.dflow INFO: Standard output for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/09be60e8-3e8b-454c-805f-c64a47f62445_characterizeImage_1124710_53.stdout parsl.dataflow.dflow INFO: Standard error for task 418 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/09be60e8-3e8b-454c-805f-c64a47f62445_characterizeImage_1124710_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:01 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 11 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b695129f77563daad338b9ae5819cde with result from task 11 parsl.dataflow.dflow DEBUG: Task 11 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/b8e4b587-45d1-40ca-8330-81fb95068ca6_calibrate_1124831_5.stdout parsl.dataflow.dflow INFO: Standard error for task 11 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/b8e4b587-45d1-40ca-8330-81fb95068ca6_calibrate_1124831_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:02 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 229 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d1dca2ead926b8f95bcc66700d70478 with result from task 229 parsl.dataflow.dflow DEBUG: Task 229 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 230 has memoization hash 829aa4d6afe9ad6174eb426e9c877cca parsl.dataflow.memoization INFO: Task 230 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adafec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 230 try 0 launched on executor multi with executor id 774 parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7ca095b8-4393-4df9-b529-12f8f355620d_calibrate_1124710_50.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7ca095b8-4393-4df9-b529-12f8f355620d_calibrate_1124710_50.stderr parsl.dataflow.dflow INFO: Standard output for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d8d1c4e2-d524-4e4a-b374-91fa828434de_characterizeImage_1124710_50.stdout parsl.dataflow.dflow INFO: Standard error for task 229 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d8d1c4e2-d524-4e4a-b374-91fa828434de_characterizeImage_1124710_50.stderr parsl.dataflow.dflow INFO: Task 853 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f6f2f9406a3a9ea8e88263cbb7e597f6 with result from task 853 parsl.dataflow.dflow DEBUG: Task 853 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 854 has memoization hash 47fa0a130cd4c470960bf72dfea7c6c0 parsl.dataflow.memoization INFO: Task 854 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 854 try 0 launched on executor multi with executor id 775 parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/7bb2b433-1a01-4454-858f-9c787e55fdc2_calibrate_1124828_44.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/7bb2b433-1a01-4454-858f-9c787e55fdc2_calibrate_1124828_44.stderr parsl.dataflow.dflow INFO: Standard output for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/2378906a-8096-4c74-b050-bebab7e32796_characterizeImage_1124828_44.stdout parsl.dataflow.dflow INFO: Standard error for task 853 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/2378906a-8096-4c74-b050-bebab7e32796_characterizeImage_1124828_44.stderr parsl.dataflow.dflow INFO: Task 535 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 20706a80c48bca708dbeb27289b9d5d0 with result from task 535 parsl.dataflow.dflow DEBUG: Task 535 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 536 has memoization hash 3e1963aa48ec7f4d76c38f8861721e7a parsl.dataflow.memoization INFO: Task 536 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529853f60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 536 try 0 launched on executor multi with executor id 776 parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/28c36749-9191-4bfb-90de-3edf776c1d76_calibrate_1124707_12.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/28c36749-9191-4bfb-90de-3edf776c1d76_calibrate_1124707_12.stderr parsl.dataflow.dflow INFO: Standard output for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/9ba0b0b7-31a3-4fa1-8023-f1b0553970cd_characterizeImage_1124707_12.stdout parsl.dataflow.dflow INFO: Standard error for task 535 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/9ba0b0b7-31a3-4fa1-8023-f1b0553970cd_characterizeImage_1124707_12.stderr parsl.dataflow.dflow INFO: Task 667 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dbbd5a6985a7fdf6b22b121ffc161c5 with result from task 667 parsl.dataflow.dflow DEBUG: Task 667 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 668 has memoization hash 01aa482d5bd573e4fbd6d8889c32cb8c parsl.dataflow.memoization INFO: Task 668 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 668 try 0 launched on executor multi with executor id 777 parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/e37aaa39-5822-41c3-a95d-aa28b1a70791_calibrate_1124704_62.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/e37aaa39-5822-41c3-a95d-aa28b1a70791_calibrate_1124704_62.stderr parsl.dataflow.dflow INFO: Standard output for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f98f063c-c4ed-4aaa-8a55-38dfbbc9a293_characterizeImage_1124704_62.stdout parsl.dataflow.dflow INFO: Standard error for task 667 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f98f063c-c4ed-4aaa-8a55-38dfbbc9a293_characterizeImage_1124704_62.stderr parsl.dataflow.dflow INFO: Task 874 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d60712c702bf109c1ab970fb060c687 with result from task 874 parsl.dataflow.dflow DEBUG: Task 874 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 875 has memoization hash 265e1149101cf1be376f0b23dd22c23a parsl.dataflow.memoization INFO: Task 875 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960a5c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 875 try 0 launched on executor multi with executor id 778 parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/2900821b-bb9a-44a0-ac2f-cac8c4927755_calibrate_1124831_38.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/2900821b-bb9a-44a0-ac2f-cac8c4927755_calibrate_1124831_38.stderr parsl.dataflow.dflow INFO: Standard output for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/72d6d45d-3aa4-47d2-9ffd-af56e4ed7198_characterizeImage_1124831_38.stdout parsl.dataflow.dflow INFO: Standard error for task 874 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/72d6d45d-3aa4-47d2-9ffd-af56e4ed7198_characterizeImage_1124831_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:03 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 466 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24435e1f5300bdbb6dc16e1ddae5358d with result from task 466 parsl.dataflow.dflow DEBUG: Task 466 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 467 has memoization hash d0076b71bd215121b76d562b53336b70 parsl.dataflow.memoization INFO: Task 467 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981f560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 467 try 0 launched on executor multi with executor id 779 parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/fe434a0b-af41-4be4-9d2f-9ec19d778168_calibrate_1124831_56.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/fe434a0b-af41-4be4-9d2f-9ec19d778168_calibrate_1124831_56.stderr parsl.dataflow.dflow INFO: Standard output for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/617bf134-50b1-4661-80ee-2ce607a16ca5_characterizeImage_1124831_56.stdout parsl.dataflow.dflow INFO: Standard error for task 466 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/617bf134-50b1-4661-80ee-2ce607a16ca5_characterizeImage_1124831_56.stderr parsl.dataflow.dflow INFO: Task 574 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 317a958e68b41741aef10699e96a7e89 with result from task 574 parsl.dataflow.dflow DEBUG: Task 574 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 575 has memoization hash 96a420cbf075152ae21b823ef0cf3612 parsl.dataflow.memoization INFO: Task 575 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969ec00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 575 try 0 launched on executor multi with executor id 780 parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/7ee0e280-24cc-4759-954f-f23974918725_calibrate_1124704_42.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/7ee0e280-24cc-4759-954f-f23974918725_calibrate_1124704_42.stderr parsl.dataflow.dflow INFO: Standard output for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/6c92ec33-3131-4263-a109-38303ba77a06_characterizeImage_1124704_42.stdout parsl.dataflow.dflow INFO: Standard error for task 574 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/6c92ec33-3131-4263-a109-38303ba77a06_characterizeImage_1124704_42.stderr parsl.dataflow.dflow INFO: Task 31 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b9edca1a26b0bdd30eed7a9d3664564 with result from task 31 parsl.dataflow.dflow DEBUG: Task 31 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 32 has memoization hash d55c274452fc726213cedb01ae3052a4 parsl.dataflow.memoization INFO: Task 32 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4b420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 32 try 0 launched on executor multi with executor id 781 parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/2b879117-db26-4824-9c43-2ee3a76a7559_calibrate_1124710_16.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/2b879117-db26-4824-9c43-2ee3a76a7559_calibrate_1124710_16.stderr parsl.dataflow.dflow INFO: Standard output for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b43c61cd-8889-4302-ac9a-90553ff45d68_characterizeImage_1124710_16.stdout parsl.dataflow.dflow INFO: Standard error for task 31 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b43c61cd-8889-4302-ac9a-90553ff45d68_characterizeImage_1124710_16.stderr parsl.dataflow.dflow INFO: Task 88 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14a951735d77640424df1a16325ba460 with result from task 88 parsl.dataflow.dflow DEBUG: Task 88 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 89 has memoization hash d0776adf3864b657d3130fea6099dde0 parsl.dataflow.memoization INFO: Task 89 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad66c00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 89 try 0 launched on executor multi with executor id 782 parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/6a58f80e-b0f2-41d5-9be7-3d05699fd27c_calibrate_1124710_41.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/6a58f80e-b0f2-41d5-9be7-3d05699fd27c_calibrate_1124710_41.stderr parsl.dataflow.dflow INFO: Standard output for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f373ed55-c87b-4204-a8ed-486f33d8a648_characterizeImage_1124710_41.stdout parsl.dataflow.dflow INFO: Standard error for task 88 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f373ed55-c87b-4204-a8ed-486f33d8a648_characterizeImage_1124710_41.stderr parsl.dataflow.dflow INFO: Task 112 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6b1ab2d18e319628740c155dae6c3c5b with result from task 112 parsl.dataflow.dflow DEBUG: Task 112 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 113 has memoization hash c946a2dcefc54baf10d60e5501b95b46 parsl.dataflow.memoization INFO: Task 113 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8c540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 113 try 0 launched on executor multi with executor id 783 parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ea75c6e3-82ba-47a6-a509-54ee2af60caa_calibrate_1124704_49.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ea75c6e3-82ba-47a6-a509-54ee2af60caa_calibrate_1124704_49.stderr parsl.dataflow.dflow INFO: Standard output for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7b0890ed-0e8a-4308-bb96-c85af93aa523_characterizeImage_1124704_49.stdout parsl.dataflow.dflow INFO: Standard error for task 112 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7b0890ed-0e8a-4308-bb96-c85af93aa523_characterizeImage_1124704_49.stderr parsl.dataflow.dflow INFO: Task 859 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e986b3f23f00955fa3496ff503253da with result from task 859 parsl.dataflow.dflow DEBUG: Task 859 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 860 has memoization hash b44466eb37f8671b06bdd3388fe91459 parsl.dataflow.memoization INFO: Task 860 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529609620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 860 try 0 launched on executor multi with executor id 784 parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/c1df79fd-1d14-4699-941f-0c366b7d263a_calibrate_1124828_52.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/c1df79fd-1d14-4699-941f-0c366b7d263a_calibrate_1124828_52.stderr parsl.dataflow.dflow INFO: Standard output for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/767d8902-f0c6-46d3-9e1e-cd8974a46f4b_characterizeImage_1124828_52.stdout parsl.dataflow.dflow INFO: Standard error for task 859 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/767d8902-f0c6-46d3-9e1e-cd8974a46f4b_characterizeImage_1124828_52.stderr parsl.dataflow.dflow INFO: Task 232 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61b0193de39b5a4d5e973f3b3dec1bd0 with result from task 232 parsl.dataflow.dflow DEBUG: Task 232 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 233 has memoization hash 38f7c2a4cbc6952d8547d46b32655233 parsl.dataflow.memoization INFO: Task 233 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a82c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 233 try 0 launched on executor multi with executor id 785 parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/d61171d0-282a-4567-9f9d-bda7d46583b8_calibrate_1124710_6.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/d61171d0-282a-4567-9f9d-bda7d46583b8_calibrate_1124710_6.stderr parsl.dataflow.dflow INFO: Standard output for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/efb7deec-80ec-4290-9d91-8cfe381fce75_characterizeImage_1124710_6.stdout parsl.dataflow.dflow INFO: Standard error for task 232 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/efb7deec-80ec-4290-9d91-8cfe381fce75_characterizeImage_1124710_6.stderr parsl.dataflow.dflow INFO: Task 403 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c861a745cbca9e9a5b8317c5535fd442 with result from task 403 parsl.dataflow.dflow DEBUG: Task 403 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 404 has memoization hash 3472590723ddb31f0029576a01a5dd10 parsl.dataflow.memoization INFO: Task 404 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ead40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 404 try 0 launched on executor multi with executor id 786 parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/f1a978f4-a814-41d2-aac1-566216e00455_calibrate_1124707_15.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/f1a978f4-a814-41d2-aac1-566216e00455_calibrate_1124707_15.stderr parsl.dataflow.dflow INFO: Standard output for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/6eea15c4-6086-48b5-916b-f75b523bc14d_characterizeImage_1124707_15.stdout parsl.dataflow.dflow INFO: Standard error for task 403 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/6eea15c4-6086-48b5-916b-f75b523bc14d_characterizeImage_1124707_15.stderr parsl.dataflow.dflow INFO: Task 323 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8e186211c167845d36a83300f217d6a with result from task 323 parsl.dataflow.dflow DEBUG: Task 323 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/e7f93319-d94d-48f7-afa8-acfb5acfebe0_calibrate_1124831_36.stdout parsl.dataflow.dflow INFO: Standard error for task 323 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/e7f93319-d94d-48f7-afa8-acfb5acfebe0_calibrate_1124831_36.stderr parsl.dataflow.dflow INFO: Task 634 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 477e5023e7581ab198be3ccba8aa8e84 with result from task 634 parsl.dataflow.dflow DEBUG: Task 634 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 635 has memoization hash 956aa95406b5eef9473c9b8af096eca5 parsl.dataflow.memoization INFO: Task 635 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e9e40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 635 try 0 launched on executor multi with executor id 787 parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/bff99b4a-25f1-4b14-80a9-411e53b57b8e_calibrate_1124710_21.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/bff99b4a-25f1-4b14-80a9-411e53b57b8e_calibrate_1124710_21.stderr parsl.dataflow.dflow INFO: Standard output for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/6d7a3e7d-54dc-420d-beb2-d4927271ff3b_characterizeImage_1124710_21.stdout parsl.dataflow.dflow INFO: Standard error for task 634 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/6d7a3e7d-54dc-420d-beb2-d4927271ff3b_characterizeImage_1124710_21.stderr parsl.dataflow.dflow INFO: Task 101 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 264aa7c36c9ef9627a379788d67bc1b2 with result from task 101 parsl.dataflow.dflow DEBUG: Task 101 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/ea400acc-1cfe-4054-ac8a-b92e473e398e_calibrate_1124831_27.stdout parsl.dataflow.dflow INFO: Standard error for task 101 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/ea400acc-1cfe-4054-ac8a-b92e473e398e_calibrate_1124831_27.stderr parsl.dataflow.dflow INFO: Task 463 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb4c2b58007ffe57d8285e00fdb17cca with result from task 463 parsl.dataflow.dflow DEBUG: Task 463 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 464 has memoization hash 5be11f1b0a164ef3c9580f4d96822cdc parsl.dataflow.memoization INFO: Task 464 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 464 try 0 launched on executor multi with executor id 788 parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/0676aaba-6bc0-4a75-9faf-bed440b7d55c_calibrate_1124831_46.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/0676aaba-6bc0-4a75-9faf-bed440b7d55c_calibrate_1124831_46.stderr parsl.dataflow.dflow INFO: Standard output for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/bb2bef86-156e-4b6e-be1a-94a3f7deb1ea_characterizeImage_1124831_46.stdout parsl.dataflow.dflow INFO: Standard error for task 463 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/bb2bef86-156e-4b6e-be1a-94a3f7deb1ea_characterizeImage_1124831_46.stderr parsl.dataflow.dflow INFO: Task 305 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44dc80a8f45b12843c30de8048a5a913 with result from task 305 parsl.dataflow.dflow DEBUG: Task 305 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/9ba2305e-e42f-498a-824b-a9750c519986_calibrate_1124831_14.stdout parsl.dataflow.dflow INFO: Standard error for task 305 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/9ba2305e-e42f-498a-824b-a9750c519986_calibrate_1124831_14.stderr parsl.dataflow.dflow INFO: Task 490 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b7aaaa66fc9eec519d39aeed3825ad75 with result from task 490 parsl.dataflow.dflow DEBUG: Task 490 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 491 has memoization hash 14095883208391a74e952a8c8d34cda3 parsl.dataflow.memoization INFO: Task 491 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529850c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 491 try 0 launched on executor multi with executor id 789 parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3bace157-5893-430f-8979-91b5084460f8_calibrate_1124831_32.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3bace157-5893-430f-8979-91b5084460f8_calibrate_1124831_32.stderr parsl.dataflow.dflow INFO: Standard output for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f7d5e89a-6d50-479b-bfa7-697ceafdf224_characterizeImage_1124831_32.stdout parsl.dataflow.dflow INFO: Standard error for task 490 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f7d5e89a-6d50-479b-bfa7-697ceafdf224_characterizeImage_1124831_32.stderr parsl.dataflow.dflow INFO: Task 343 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 306fb6e3289a5dd11b8d1ed295a94ed6 with result from task 343 parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:05 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow DEBUG: Task 343 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 344 has memoization hash 94bcdf27701a6ed9be3908e2bb6fec0d parsl.dataflow.memoization INFO: Task 344 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cf6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 344 try 0 launched on executor multi with executor id 790 parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/295e2d70-ee07-482f-9da6-93f3ef09bfbe_calibrate_1124707_38.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/295e2d70-ee07-482f-9da6-93f3ef09bfbe_calibrate_1124707_38.stderr parsl.dataflow.dflow INFO: Standard output for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/d0512725-cbda-42fe-bf92-4091d80f1a1c_characterizeImage_1124707_38.stdout parsl.dataflow.dflow INFO: Standard error for task 343 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/d0512725-cbda-42fe-bf92-4091d80f1a1c_characterizeImage_1124707_38.stderr parsl.dataflow.dflow INFO: Task 286 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffd4b71955f6eaba8ba49d8ea3a59c22 with result from task 286 parsl.dataflow.dflow DEBUG: Task 286 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 287 has memoization hash 24fc6b09138a9d93c010c39e57b35839 parsl.dataflow.memoization INFO: Task 287 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297abb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 287 try 0 launched on executor multi with executor id 791 parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/994626c2-63ce-4770-b229-c08990b995a1_calibrate_1124828_15.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/994626c2-63ce-4770-b229-c08990b995a1_calibrate_1124828_15.stderr parsl.dataflow.dflow INFO: Standard output for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/24d13a3e-f8cc-437b-8deb-16759d06ac5f_characterizeImage_1124828_15.stdout parsl.dataflow.dflow INFO: Standard error for task 286 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/24d13a3e-f8cc-437b-8deb-16759d06ac5f_characterizeImage_1124828_15.stderr parsl.dataflow.dflow INFO: Task 724 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ffe23b9c905fb1fcecfd21486cc4c2a8 with result from task 724 parsl.dataflow.dflow DEBUG: Task 724 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 725 has memoization hash 5c9f71326ebfe5c278348db2b47e659c parsl.dataflow.memoization INFO: Task 725 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977c720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 725 try 0 launched on executor multi with executor id 792 parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/1c57f535-10e1-46aa-8a93-93e2dd9a7820_calibrate_1124704_5.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/1c57f535-10e1-46aa-8a93-93e2dd9a7820_calibrate_1124704_5.stderr parsl.dataflow.dflow INFO: Standard output for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bdb5162a-f9e6-43b6-b996-95f0f68b1710_characterizeImage_1124704_5.stdout parsl.dataflow.dflow INFO: Standard error for task 724 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bdb5162a-f9e6-43b6-b996-95f0f68b1710_characterizeImage_1124704_5.stderr parsl.dataflow.dflow INFO: Task 778 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 940501a21a0b14b20d64c638e63639c0 with result from task 778 parsl.dataflow.dflow DEBUG: Task 778 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 779 has memoization hash 0a4c92645a54c683fdccfd2e31191600 parsl.dataflow.memoization INFO: Task 779 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c8360> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 779 try 0 launched on executor multi with executor id 793 parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/651bbdb2-a798-4596-8296-7942fa3c1c40_calibrate_1124828_3.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/651bbdb2-a798-4596-8296-7942fa3c1c40_calibrate_1124828_3.stderr parsl.dataflow.dflow INFO: Standard output for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/28d32b11-7189-4984-a945-5ef0291defb4_characterizeImage_1124828_3.stdout parsl.dataflow.dflow INFO: Standard error for task 778 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/28d32b11-7189-4984-a945-5ef0291defb4_characterizeImage_1124828_3.stderr parsl.dataflow.dflow INFO: Task 703 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6ace7840b88ca7ab44b5900b8cd83de with result from task 703 parsl.dataflow.dflow DEBUG: Task 703 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 704 has memoization hash 7876dafe26d3608f8ce73f2826fc2d38 parsl.dataflow.memoization INFO: Task 704 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973b6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 704 try 0 launched on executor multi with executor id 794 parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/fe036f83-95fc-4c75-a70d-cc8841d35c66_calibrate_1124828_19.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/fe036f83-95fc-4c75-a70d-cc8841d35c66_calibrate_1124828_19.stderr parsl.dataflow.dflow INFO: Standard output for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a9a96111-ea1a-4196-879c-508c571bcdfa_characterizeImage_1124828_19.stdout parsl.dataflow.dflow INFO: Standard error for task 703 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/a9a96111-ea1a-4196-879c-508c571bcdfa_characterizeImage_1124828_19.stderr parsl.dataflow.dflow INFO: Task 151 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05f4066d889125d62cba8d48b72de52d with result from task 151 parsl.dataflow.dflow DEBUG: Task 151 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 152 has memoization hash b0f2fee52eacd2f4d0dd606d68f60404 parsl.dataflow.memoization INFO: Task 152 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8ede0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 152 try 0 launched on executor multi with executor id 795 parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a9c1bfc0-7cfd-4aa5-8d67-690a85ad8f6a_calibrate_1124828_26.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a9c1bfc0-7cfd-4aa5-8d67-690a85ad8f6a_calibrate_1124828_26.stderr parsl.dataflow.dflow INFO: Standard output for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/1f17d930-23b9-4627-853c-70a00da4d041_characterizeImage_1124828_26.stdout parsl.dataflow.dflow INFO: Standard error for task 151 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/1f17d930-23b9-4627-853c-70a00da4d041_characterizeImage_1124828_26.stderr parsl.dataflow.dflow INFO: Task 379 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a27504cde7359b916a3aa0ad9be5ac11 with result from task 379 parsl.dataflow.dflow DEBUG: Task 379 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 380 has memoization hash 51cc15169b39d8a2232564c47af8daf2 parsl.dataflow.memoization INFO: Task 380 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ebec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 380 try 0 launched on executor multi with executor id 796 parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ebdd4b7c-08d0-43ef-b1b1-6c9f71c656df_calibrate_1124704_45.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ebdd4b7c-08d0-43ef-b1b1-6c9f71c656df_calibrate_1124704_45.stderr parsl.dataflow.dflow INFO: Standard output for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/59aba1fe-a453-4fcc-90f5-87c5611a72e3_characterizeImage_1124704_45.stdout parsl.dataflow.dflow INFO: Standard error for task 379 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/59aba1fe-a453-4fcc-90f5-87c5611a72e3_characterizeImage_1124704_45.stderr parsl.dataflow.dflow INFO: Task 505 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2fc1b33a37a956cd28eaf34fb9c728b0 with result from task 505 parsl.dataflow.dflow DEBUG: Task 505 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 506 has memoization hash 8b61c2943b18fb9f21bbc6b5a4c7273b parsl.dataflow.memoization INFO: Task 506 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529851260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 506 try 0 launched on executor multi with executor id 797 parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/73c83ca9-42d0-40b3-afb0-e5dfcf2ca78d_calibrate_1124831_4.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/73c83ca9-42d0-40b3-afb0-e5dfcf2ca78d_calibrate_1124831_4.stderr parsl.dataflow.dflow INFO: Standard output for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/1b45cdee-18d5-4284-82ae-de8e8a6a1b16_characterizeImage_1124831_4.stdout parsl.dataflow.dflow INFO: Standard error for task 505 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/1b45cdee-18d5-4284-82ae-de8e8a6a1b16_characterizeImage_1124831_4.stderr parsl.dataflow.dflow INFO: Task 196 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 709c05debb8de0e6d38188cf825b1aa2 with result from task 196 parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:06 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow DEBUG: Task 196 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 197 has memoization hash 359b6e194f6bd54372a50902fc8770d5 parsl.dataflow.memoization INFO: Task 197 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adadd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 197 try 0 launched on executor multi with executor id 798 parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/f9f424a7-bff6-4c43-b6ae-4682f72c2540_calibrate_1124831_34.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/f9f424a7-bff6-4c43-b6ae-4682f72c2540_calibrate_1124831_34.stderr parsl.dataflow.dflow INFO: Standard output for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/fce7f302-3289-437e-8b70-69c204d4ed7b_characterizeImage_1124831_34.stdout parsl.dataflow.dflow INFO: Standard error for task 196 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/fce7f302-3289-437e-8b70-69c204d4ed7b_characterizeImage_1124831_34.stderr parsl.dataflow.dflow INFO: Task 409 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e00424e7e1c0f46fc40e86f265b5d56d with result from task 409 parsl.dataflow.dflow DEBUG: Task 409 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 410 has memoization hash cc9406f90c86ef6eae25651434f6546c parsl.dataflow.memoization INFO: Task 410 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297eb6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 410 try 0 launched on executor multi with executor id 799 parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/35c94c10-32b6-4796-907c-8bfc0119f8f0_calibrate_1124707_48.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/35c94c10-32b6-4796-907c-8bfc0119f8f0_calibrate_1124707_48.stderr parsl.dataflow.dflow INFO: Standard output for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/fdb91c8c-087c-44ad-94b3-4603e22ed44a_characterizeImage_1124707_48.stdout parsl.dataflow.dflow INFO: Standard error for task 409 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/fdb91c8c-087c-44ad-94b3-4603e22ed44a_characterizeImage_1124707_48.stderr parsl.dataflow.dflow INFO: Task 877 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54869dc7fe91dbe123fde3f5c16f7d7d with result from task 877 parsl.dataflow.dflow DEBUG: Task 877 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 878 has memoization hash 4b1f72c0df23bf72102bc8c1df2a9e2a parsl.dataflow.memoization INFO: Task 878 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 878 try 0 launched on executor multi with executor id 800 parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/5d2869b7-0aba-4f58-b9e8-361e5c7fd2d3_calibrate_1124828_12.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/5d2869b7-0aba-4f58-b9e8-361e5c7fd2d3_calibrate_1124828_12.stderr parsl.dataflow.dflow INFO: Standard output for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/065861c3-2b9a-4f85-9eb7-d1795ecdbbf1_characterizeImage_1124828_12.stdout parsl.dataflow.dflow INFO: Standard error for task 877 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/065861c3-2b9a-4f85-9eb7-d1795ecdbbf1_characterizeImage_1124828_12.stderr parsl.dataflow.dflow INFO: Task 148 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry af7078931362430c357994d32ed0989c with result from task 148 parsl.dataflow.dflow DEBUG: Task 148 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 149 has memoization hash f0cadfc89c98672f1be71deb256abdec parsl.dataflow.memoization INFO: Task 149 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 149 try 0 launched on executor multi with executor id 801 parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4f4d3843-8adf-4155-9d57-84bdffc92437_calibrate_1124831_21.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4f4d3843-8adf-4155-9d57-84bdffc92437_calibrate_1124831_21.stderr parsl.dataflow.dflow INFO: Standard output for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/fdf62589-747b-4c4c-86ae-381591f93560_characterizeImage_1124831_21.stdout parsl.dataflow.dflow INFO: Standard error for task 148 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/fdf62589-747b-4c4c-86ae-381591f93560_characterizeImage_1124831_21.stderr parsl.dataflow.dflow INFO: Task 338 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a3e2897a85cbfe7700255b1e15a1aa2d with result from task 338 parsl.dataflow.dflow DEBUG: Task 338 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/51596cec-fbde-4e06-b45a-545dc377a4fd_calibrate_1124828_31.stdout parsl.dataflow.dflow INFO: Standard error for task 338 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/51596cec-fbde-4e06-b45a-545dc377a4fd_calibrate_1124828_31.stderr parsl.dataflow.dflow INFO: Task 679 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f1ad4969f921e2a06e3bcd8cf2348940 with result from task 679 parsl.dataflow.dflow DEBUG: Task 679 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 680 has memoization hash 4cb6230821350dc7c9f9451618364e2b parsl.dataflow.memoization INFO: Task 680 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297396c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 680 try 0 launched on executor multi with executor id 802 parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/152cabc0-8fcb-41f6-9a2c-2f69097d5b8d_calibrate_1124710_3.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/152cabc0-8fcb-41f6-9a2c-2f69097d5b8d_calibrate_1124710_3.stderr parsl.dataflow.dflow INFO: Standard output for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e413339b-23d9-4807-842b-f988cd70a5c6_characterizeImage_1124710_3.stdout parsl.dataflow.dflow INFO: Standard error for task 679 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e413339b-23d9-4807-842b-f988cd70a5c6_characterizeImage_1124710_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:07 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 68 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b79270740dc1be70f6950f556135ac88 with result from task 68 parsl.dataflow.dflow DEBUG: Task 68 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f55bd7e2-76bb-4368-9a07-09af0bf3b95c_calibrate_1124828_59.stdout parsl.dataflow.dflow INFO: Standard error for task 68 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f55bd7e2-76bb-4368-9a07-09af0bf3b95c_calibrate_1124828_59.stderr parsl.dataflow.dflow INFO: Task 560 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8758af19badc1dc49176f87032743792 with result from task 560 parsl.dataflow.dflow DEBUG: Task 560 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a1cf385e-d375-4f93-b8c8-c0e0a92888c0_calibrate_1124831_9.stdout parsl.dataflow.dflow INFO: Standard error for task 560 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a1cf385e-d375-4f93-b8c8-c0e0a92888c0_calibrate_1124831_9.stderr parsl.dataflow.dflow INFO: Task 883 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b595716e5928ed9893c40b90aa63140 with result from task 883 parsl.dataflow.dflow DEBUG: Task 883 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 884 has memoization hash a8c1943fbc76798a70446079045ff5f1 parsl.dataflow.memoization INFO: Task 884 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960af20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 884 try 0 launched on executor multi with executor id 803 parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/ad35a760-797d-42b7-909a-fda183dc5e26_calibrate_1124828_22.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/ad35a760-797d-42b7-909a-fda183dc5e26_calibrate_1124828_22.stderr parsl.dataflow.dflow INFO: Standard output for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/fca9eacd-7210-45c5-8537-632f5f202710_characterizeImage_1124828_22.stdout parsl.dataflow.dflow INFO: Standard error for task 883 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/fca9eacd-7210-45c5-8537-632f5f202710_characterizeImage_1124828_22.stderr parsl.dataflow.dflow INFO: Task 71 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f40f186add58351ad52e6f8902e9a02e with result from task 71 parsl.dataflow.dflow DEBUG: Task 71 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/45ca6e14-e625-42fe-b653-ff41074de4d7_calibrate_1124831_47.stdout parsl.dataflow.dflow INFO: Standard error for task 71 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/45ca6e14-e625-42fe-b653-ff41074de4d7_calibrate_1124831_47.stderr parsl.dataflow.dflow INFO: Task 428 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbf96f9cc5aac16b976eadbd1a49ab11 with result from task 428 parsl.dataflow.dflow DEBUG: Task 428 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/1a37c963-4a78-43af-9394-f19568fa7c20_calibrate_1124831_60.stdout parsl.dataflow.dflow INFO: Standard error for task 428 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/1a37c963-4a78-43af-9394-f19568fa7c20_calibrate_1124831_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:08 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 314 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a506dd989d7bbe3e77919ef1f4ebb96 with result from task 314 parsl.dataflow.dflow DEBUG: Task 314 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3bac62ac-b48f-4bba-a73d-cad60cbb8c0e_calibrate_1124831_1.stdout parsl.dataflow.dflow INFO: Standard error for task 314 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3bac62ac-b48f-4bba-a73d-cad60cbb8c0e_calibrate_1124831_1.stderr parsl.dataflow.dflow INFO: Task 47 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcd35d136d0f9dad871b9c448f5fcedd with result from task 47 parsl.dataflow.dflow DEBUG: Task 47 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/cb3db7b8-caa3-438a-bcd3-91429c967614_calibrate_1124831_33.stdout parsl.dataflow.dflow INFO: Standard error for task 47 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/cb3db7b8-caa3-438a-bcd3-91429c967614_calibrate_1124831_33.stderr parsl.dataflow.dflow INFO: Task 788 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d13d456aa913c9bfef06fa675d4481cd with result from task 788 parsl.dataflow.dflow DEBUG: Task 788 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f9b21050-79f3-47e0-ad78-23f149f6e6cd_calibrate_1124828_5.stdout parsl.dataflow.dflow INFO: Standard error for task 788 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f9b21050-79f3-47e0-ad78-23f149f6e6cd_calibrate_1124828_5.stderr parsl.dataflow.dflow INFO: Task 77 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b225a98483049dee3a3cf1afd82e35e with result from task 77 parsl.dataflow.dflow DEBUG: Task 77 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/fb86883b-98f7-4453-9732-7a4e9ab44ec7_calibrate_1124831_30.stdout parsl.dataflow.dflow INFO: Standard error for task 77 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/fb86883b-98f7-4453-9732-7a4e9ab44ec7_calibrate_1124831_30.stderr parsl.dataflow.dflow INFO: Task 664 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16b8f61ba9e1d1f20187aa04eabde6bd with result from task 664 parsl.dataflow.dflow DEBUG: Task 664 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 665 has memoization hash 3c78d426756a518d01c53a6dd76c21f4 parsl.dataflow.memoization INFO: Task 665 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973bec0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 665 try 0 launched on executor multi with executor id 804 parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/aa713c1b-1853-44a0-bb76-ffd5b9972b76_calibrate_1124710_39.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/aa713c1b-1853-44a0-bb76-ffd5b9972b76_calibrate_1124710_39.stderr parsl.dataflow.dflow INFO: Standard output for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b7ddbbeb-c55b-437f-8db6-108f505fe4be_characterizeImage_1124710_39.stdout parsl.dataflow.dflow INFO: Standard error for task 664 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b7ddbbeb-c55b-437f-8db6-108f505fe4be_characterizeImage_1124710_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:09 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 415 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c91191510c1daec301e5bd6be96f1dd with result from task 415 parsl.dataflow.dflow DEBUG: Task 415 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 416 has memoization hash bb21dea1fc3efa82ea8fa99a74f0ff72 parsl.dataflow.memoization INFO: Task 416 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e9bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 416 try 0 launched on executor multi with executor id 805 parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/29cbee83-4aae-469d-a1bd-6a1045bc2637_calibrate_1124707_9.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/29cbee83-4aae-469d-a1bd-6a1045bc2637_calibrate_1124707_9.stderr parsl.dataflow.dflow INFO: Standard output for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/c1ec030a-0e8d-4a22-bbc7-444d6cea8319_characterizeImage_1124707_9.stdout parsl.dataflow.dflow INFO: Standard error for task 415 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/c1ec030a-0e8d-4a22-bbc7-444d6cea8319_characterizeImage_1124707_9.stderr parsl.dataflow.dflow INFO: Task 238 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90a097703cd912fe43b7a01b189e7332 with result from task 238 parsl.dataflow.dflow DEBUG: Task 238 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 239 has memoization hash 34ea410ba8141b401779d65cac8cdeb9 parsl.dataflow.memoization INFO: Task 239 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a8900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 239 try 0 launched on executor multi with executor id 806 parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1aba9bad-8c9d-456b-b3f2-1f4cab88b235_calibrate_1124710_17.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1aba9bad-8c9d-456b-b3f2-1f4cab88b235_calibrate_1124710_17.stderr parsl.dataflow.dflow INFO: Standard output for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/57414bd4-8170-4c5f-a333-8740a2c0fdc8_characterizeImage_1124710_17.stdout parsl.dataflow.dflow INFO: Standard error for task 238 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/57414bd4-8170-4c5f-a333-8740a2c0fdc8_characterizeImage_1124710_17.stderr parsl.dataflow.dflow INFO: Task 319 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d76112bf5a8875caad5c2f8bac39d7ef with result from task 319 parsl.dataflow.dflow DEBUG: Task 319 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 320 has memoization hash 44a195675c74d52022cbb30aed18ea81 parsl.dataflow.memoization INFO: Task 320 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cdda0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 320 try 0 launched on executor multi with executor id 807 parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/fab2a62b-e52b-41de-a3fa-434345661961_calibrate_1124828_18.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/fab2a62b-e52b-41de-a3fa-434345661961_calibrate_1124828_18.stderr parsl.dataflow.dflow INFO: Standard output for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/b4531e41-c441-4a11-8be8-67b766171bd0_characterizeImage_1124828_18.stdout parsl.dataflow.dflow INFO: Standard error for task 319 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/b4531e41-c441-4a11-8be8-67b766171bd0_characterizeImage_1124828_18.stderr parsl.dataflow.dflow INFO: Task 925 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 828b317b5ed21103f60753422048e912 with result from task 925 parsl.dataflow.dflow DEBUG: Task 925 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 926 has memoization hash ac3f67e7a9ebfcdfa0b04431dbd2a052 parsl.dataflow.memoization INFO: Task 926 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296653a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 926 try 0 launched on executor multi with executor id 808 parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/6e868246-72d0-457e-86fd-4541d31bde82_calibrate_1124828_7.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/6e868246-72d0-457e-86fd-4541d31bde82_calibrate_1124828_7.stderr parsl.dataflow.dflow INFO: Standard output for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/f706409f-d380-4283-a729-1d6e68dd6bfe_characterizeImage_1124828_7.stdout parsl.dataflow.dflow INFO: Standard error for task 925 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/f706409f-d380-4283-a729-1d6e68dd6bfe_characterizeImage_1124828_7.stderr parsl.dataflow.dflow INFO: Task 220 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15ba0bec44e03fa2357e6f71e7502d03 with result from task 220 parsl.dataflow.dflow DEBUG: Task 220 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 221 has memoization hash 49423f01bd68bc26df48ae9e14b07391 parsl.dataflow.memoization INFO: Task 221 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaf600> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 221 try 0 launched on executor multi with executor id 809 parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/bf836b3c-aecb-4116-b23e-f7b6d1e0a3c2_calibrate_1124828_17.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/bf836b3c-aecb-4116-b23e-f7b6d1e0a3c2_calibrate_1124828_17.stderr parsl.dataflow.dflow INFO: Standard output for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/f24a39bb-0c40-4d9f-97f8-95df8a541eb2_characterizeImage_1124828_17.stdout parsl.dataflow.dflow INFO: Standard error for task 220 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/f24a39bb-0c40-4d9f-97f8-95df8a541eb2_characterizeImage_1124828_17.stderr parsl.dataflow.dflow INFO: Task 212 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90a5dca4a4e91ce7f68f3fa7cbde2ca4 with result from task 212 parsl.dataflow.dflow DEBUG: Task 212 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8823aba8-c4c5-4c70-8b0e-c030e2c216d3_calibrate_1124828_9.stdout parsl.dataflow.dflow INFO: Standard error for task 212 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8823aba8-c4c5-4c70-8b0e-c030e2c216d3_calibrate_1124828_9.stderr parsl.dataflow.dflow INFO: Task 424 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae974ea880856a1670603e1509ecad8b with result from task 424 parsl.dataflow.dflow DEBUG: Task 424 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 425 has memoization hash 44482ec22626935f6ddc46f01a72bd09 parsl.dataflow.memoization INFO: Task 425 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981c2c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 425 try 0 launched on executor multi with executor id 810 parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/bea6868c-159a-44fb-908e-24a63c00d5e1_calibrate_1124704_41.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/bea6868c-159a-44fb-908e-24a63c00d5e1_calibrate_1124704_41.stderr parsl.dataflow.dflow INFO: Standard output for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bd31ce0e-38b1-4eac-a126-8ee5c782a896_characterizeImage_1124704_41.stdout parsl.dataflow.dflow INFO: Standard error for task 424 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/bd31ce0e-38b1-4eac-a126-8ee5c782a896_characterizeImage_1124704_41.stderr parsl.dataflow.dflow INFO: Task 469 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b240dc26005ed7a8f5f751b2c197751 with result from task 469 parsl.dataflow.dflow DEBUG: Task 469 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 470 has memoization hash 2c3913ed39677e8a6f0a3e93557e51d2 parsl.dataflow.memoization INFO: Task 470 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981f920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 470 try 0 launched on executor multi with executor id 811 parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4fa65211-1768-4381-bddf-58e0bde1f57c_calibrate_1124831_54.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4fa65211-1768-4381-bddf-58e0bde1f57c_calibrate_1124831_54.stderr parsl.dataflow.dflow INFO: Standard output for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f61880eb-cb2c-4b10-9338-b281206dae43_characterizeImage_1124831_54.stdout parsl.dataflow.dflow INFO: Standard error for task 469 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/f61880eb-cb2c-4b10-9338-b281206dae43_characterizeImage_1124831_54.stderr parsl.dataflow.dflow INFO: Task 554 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8691e1bcc463482942b125b221ada2f with result from task 554 parsl.dataflow.dflow DEBUG: Task 554 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/5305c0a0-d689-4770-9dbf-fb29774a7a28_calibrate_1124831_8.stdout parsl.dataflow.dflow INFO: Standard error for task 554 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/5305c0a0-d689-4770-9dbf-fb29774a7a28_calibrate_1124831_8.stderr parsl.dataflow.dflow INFO: Task 616 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c31175cc83f241284902b8cb4dace71f with result from task 616 parsl.dataflow.dflow DEBUG: Task 616 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 617 has memoization hash 95ba8342b3ad70fffa8f932c327c0eec parsl.dataflow.memoization INFO: Task 617 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e9080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 617 try 0 launched on executor multi with executor id 812 parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/034d316f-3e40-444d-b3d2-3280c22b756e_calibrate_1124710_58.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/034d316f-3e40-444d-b3d2-3280c22b756e_calibrate_1124710_58.stderr parsl.dataflow.dflow INFO: Standard output for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1232cf25-d352-4464-8cc6-21a6c6bcfb39_characterizeImage_1124710_58.stdout parsl.dataflow.dflow INFO: Standard error for task 616 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1232cf25-d352-4464-8cc6-21a6c6bcfb39_characterizeImage_1124710_58.stderr parsl.dataflow.dflow INFO: Task 511 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c2b337244c0ca1bec8515f1e56f89c9 with result from task 511 parsl.dataflow.dflow DEBUG: Task 511 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 512 has memoization hash 42a357a8a551d81feb79ac877d2998f3 parsl.dataflow.memoization INFO: Task 512 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 512 try 0 launched on executor multi with executor id 813 parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/004749a0-39cb-4d82-84ea-cab7ea638d88_calibrate_1124831_50.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/004749a0-39cb-4d82-84ea-cab7ea638d88_calibrate_1124831_50.stderr parsl.dataflow.dflow INFO: Standard output for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/c217ca28-1c75-46da-87fd-5941c9ff7eb8_characterizeImage_1124831_50.stdout parsl.dataflow.dflow INFO: Standard error for task 511 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/c217ca28-1c75-46da-87fd-5941c9ff7eb8_characterizeImage_1124831_50.stderr parsl.dataflow.dflow INFO: Task 706 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c61958160d0137d7d253ba0cd334aed7 with result from task 706 parsl.dataflow.dflow DEBUG: Task 706 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 707 has memoization hash 3bcbd7904d8d46109ae27da4fe848729 parsl.dataflow.memoization INFO: Task 707 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973b380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 707 try 0 launched on executor multi with executor id 814 parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/10d5a4cd-3ef6-43ce-bf62-04d517af491d_calibrate_1124831_15.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/10d5a4cd-3ef6-43ce-bf62-04d517af491d_calibrate_1124831_15.stderr parsl.dataflow.dflow INFO: Standard output for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/83d39810-cc60-414d-81ea-a3c2eb390752_characterizeImage_1124831_15.stdout parsl.dataflow.dflow INFO: Standard error for task 706 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/83d39810-cc60-414d-81ea-a3c2eb390752_characterizeImage_1124831_15.stderr parsl.dataflow.dflow INFO: Task 457 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea4a67b34158a9da41c02b323b6319f9 with result from task 457 parsl.dataflow.dflow DEBUG: Task 457 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 458 has memoization hash 09a2f6797875026d8256cd05b08a651f parsl.dataflow.memoization INFO: Task 458 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 458 try 0 launched on executor multi with executor id 815 parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/6993bf89-2b0b-457a-9284-97091d643c42_calibrate_1124828_35.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/6993bf89-2b0b-457a-9284-97091d643c42_calibrate_1124828_35.stderr parsl.dataflow.dflow INFO: Standard output for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/e0d0c77f-0784-4ed2-98c8-c223582bfbf4_characterizeImage_1124828_35.stdout parsl.dataflow.dflow INFO: Standard error for task 457 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/e0d0c77f-0784-4ed2-98c8-c223582bfbf4_characterizeImage_1124828_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:12 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 865 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47791c3f3014497d052a73ad5f62820f with result from task 865 parsl.dataflow.dflow DEBUG: Task 865 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 866 has memoization hash 8e46dab2d052b4663e42837b9edeee1e parsl.dataflow.memoization INFO: Task 866 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529609c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 866 try 0 launched on executor multi with executor id 816 parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1528c911-116f-40f6-b486-b5e3f627c5d6_calibrate_1124707_14.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1528c911-116f-40f6-b486-b5e3f627c5d6_calibrate_1124707_14.stderr parsl.dataflow.dflow INFO: Standard output for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/83d6e29e-a037-4581-a909-06a98e7099cb_characterizeImage_1124707_14.stdout parsl.dataflow.dflow INFO: Standard error for task 865 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/83d6e29e-a037-4581-a909-06a98e7099cb_characterizeImage_1124707_14.stderr parsl.dataflow.dflow INFO: Task 556 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1c19dc903a155901c61283b228ccf799 with result from task 556 parsl.dataflow.dflow DEBUG: Task 556 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 557 has memoization hash 0a742cfc9fee604c552809d39de3d6c8 parsl.dataflow.memoization INFO: Task 557 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969d080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 557 try 0 launched on executor multi with executor id 817 parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/a3637201-6b78-40b8-9b56-cf4d8706c45c_calibrate_1124704_17.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/a3637201-6b78-40b8-9b56-cf4d8706c45c_calibrate_1124704_17.stderr parsl.dataflow.dflow INFO: Standard output for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d5c23c5c-a7d1-41aa-b7d8-1366130690cb_characterizeImage_1124704_17.stdout parsl.dataflow.dflow INFO: Standard error for task 556 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d5c23c5c-a7d1-41aa-b7d8-1366130690cb_characterizeImage_1124704_17.stderr parsl.dataflow.dflow INFO: Task 62 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 010385f0374f87e56bbd2ad99afb64eb with result from task 62 parsl.dataflow.dflow DEBUG: Task 62 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/699a6afc-cec4-4df9-a07b-dbe11fdfd89f_calibrate_1124831_40.stdout parsl.dataflow.dflow INFO: Standard error for task 62 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/699a6afc-cec4-4df9-a07b-dbe11fdfd89f_calibrate_1124831_40.stderr parsl.dataflow.dflow INFO: Task 110 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8a1faaf4b0b4958f3b5c0061e36d6aa with result from task 110 parsl.dataflow.dflow DEBUG: Task 110 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/b3530352-5e91-4a44-b83e-faff8d7d98dd_calibrate_1124831_42.stdout parsl.dataflow.dflow INFO: Standard error for task 110 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/b3530352-5e91-4a44-b83e-faff8d7d98dd_calibrate_1124831_42.stderr parsl.dataflow.dflow INFO: Task 856 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fde0a0951042f1843737dc045f96e3e with result from task 856 parsl.dataflow.dflow DEBUG: Task 856 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 857 has memoization hash 7e518ecaa0faa2993c2fe1fa476df1bc parsl.dataflow.memoization INFO: Task 857 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529609300> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 857 try 0 launched on executor multi with executor id 818 parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/3d632c1a-13e8-4444-a3c3-f74f10a8282a_calibrate_1124710_5.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/3d632c1a-13e8-4444-a3c3-f74f10a8282a_calibrate_1124710_5.stderr parsl.dataflow.dflow INFO: Standard output for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/23b2cc8b-c6cb-409e-8ad1-a85945b1e7dd_characterizeImage_1124710_5.stdout parsl.dataflow.dflow INFO: Standard error for task 856 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/23b2cc8b-c6cb-409e-8ad1-a85945b1e7dd_characterizeImage_1124710_5.stderr parsl.dataflow.dflow INFO: Task 440 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a29feffff33908d06ecaa664efb507e with result from task 440 parsl.dataflow.dflow DEBUG: Task 440 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/df3eb3dc-5bb5-4541-8c28-c95ba73ffa96_calibrate_1124828_23.stdout parsl.dataflow.dflow INFO: Standard error for task 440 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/df3eb3dc-5bb5-4541-8c28-c95ba73ffa96_calibrate_1124828_23.stderr parsl.dataflow.dflow INFO: Task 595 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89210d7d95b95ed654e710f9b6d1778b with result from task 595 parsl.dataflow.dflow DEBUG: Task 595 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 596 has memoization hash c8e16394e51b97ba76da0fffecf4b50b parsl.dataflow.memoization INFO: Task 596 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296eaca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 596 try 0 launched on executor multi with executor id 819 parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/db265463-0900-41f6-b433-65962dad4b0e_calibrate_1124707_58.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/db265463-0900-41f6-b433-65962dad4b0e_calibrate_1124707_58.stderr parsl.dataflow.dflow INFO: Standard output for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/ae38fb96-1070-4562-92d8-5600f2518c05_characterizeImage_1124707_58.stdout parsl.dataflow.dflow INFO: Standard error for task 595 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/ae38fb96-1070-4562-92d8-5600f2518c05_characterizeImage_1124707_58.stderr parsl.dataflow.dflow INFO: Task 128 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bef96d85909e943687eaea34ed95c5f5 with result from task 128 parsl.dataflow.dflow DEBUG: Task 128 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/379f35c8-0a2e-4ca3-ad38-bc02f7d19b7c_calibrate_1124828_13.stdout parsl.dataflow.dflow INFO: Standard error for task 128 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/379f35c8-0a2e-4ca3-ad38-bc02f7d19b7c_calibrate_1124828_13.stderr parsl.dataflow.dflow INFO: Task 886 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89973e3adcc9f336fdcece7e8f929c91 with result from task 886 parsl.dataflow.dflow DEBUG: Task 886 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 887 has memoization hash c56454f6b4b859893ed93f226b3c2afb parsl.dataflow.memoization INFO: Task 887 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960b240> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 887 try 0 launched on executor multi with executor id 820 parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/dc676de6-2ea3-4e6a-9a9d-8994d9330848_calibrate_1124704_57.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/dc676de6-2ea3-4e6a-9a9d-8994d9330848_calibrate_1124704_57.stderr parsl.dataflow.dflow INFO: Standard output for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b02cf82c-6973-4a16-8734-78394f197f9c_characterizeImage_1124704_57.stdout parsl.dataflow.dflow INFO: Standard error for task 886 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b02cf82c-6973-4a16-8734-78394f197f9c_characterizeImage_1124704_57.stderr parsl.dataflow.dflow INFO: Task 56 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 25c0c69f928e3f2a0bbe086c88353e44 with result from task 56 parsl.dataflow.dflow DEBUG: Task 56 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/555ce685-4826-4b54-9c51-6291ea540720_calibrate_1124831_6.stdout parsl.dataflow.dflow INFO: Standard error for task 56 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/555ce685-4826-4b54-9c51-6291ea540720_calibrate_1124831_6.stderr parsl.dataflow.dflow INFO: Task 14 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dee48d2ff5223a0d610b73fb4b26c11e with result from task 14 parsl.dataflow.dflow DEBUG: Task 14 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/6d615dec-b5c8-42ce-8bce-9eb89254df50_calibrate_1124828_28.stdout parsl.dataflow.dflow INFO: Standard error for task 14 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/6d615dec-b5c8-42ce-8bce-9eb89254df50_calibrate_1124828_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:14 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 775 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d809ffc247d8b1271e1f6a7c3b65b02e with result from task 775 parsl.dataflow.dflow DEBUG: Task 775 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 776 has memoization hash be6ca0e51228ed8777d72b18b6070677 parsl.dataflow.memoization INFO: Task 776 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977ee80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 776 try 0 launched on executor multi with executor id 821 parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ff4c9252-51e9-4975-8666-2199fbb6ce33_calibrate_1124710_8.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ff4c9252-51e9-4975-8666-2199fbb6ce33_calibrate_1124710_8.stderr parsl.dataflow.dflow INFO: Standard output for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/c72d92d3-6863-4b77-87c0-c01d113e0b1c_characterizeImage_1124710_8.stdout parsl.dataflow.dflow INFO: Standard error for task 775 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/c72d92d3-6863-4b77-87c0-c01d113e0b1c_characterizeImage_1124710_8.stderr parsl.dataflow.dflow INFO: Task 95 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ece8d27881a97b860fcc2748aa2a306 with result from task 95 parsl.dataflow.dflow DEBUG: Task 95 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/e72563e3-f600-4c51-9122-cd99cf0d95b5_calibrate_1124831_23.stdout parsl.dataflow.dflow INFO: Standard error for task 95 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/e72563e3-f600-4c51-9122-cd99cf0d95b5_calibrate_1124831_23.stderr parsl.dataflow.dflow INFO: Task 550 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68db9990fa1af305e9f35a7d89bcc429 with result from task 550 parsl.dataflow.dflow DEBUG: Task 550 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 551 has memoization hash 1ca1cd6ab9e015f717cf956d08ce91c1 parsl.dataflow.memoization INFO: Task 551 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969c900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 551 try 0 launched on executor multi with executor id 822 parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/5f052231-aee6-49bd-b4e6-d0a4e08aef64_calibrate_1124704_53.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/5f052231-aee6-49bd-b4e6-d0a4e08aef64_calibrate_1124704_53.stderr parsl.dataflow.dflow INFO: Standard output for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b754b0a6-5cf5-4fa6-ae04-083027463a19_characterizeImage_1124704_53.stdout parsl.dataflow.dflow INFO: Standard error for task 550 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b754b0a6-5cf5-4fa6-ae04-083027463a19_characterizeImage_1124704_53.stderr parsl.dataflow.dflow INFO: Task 43 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b615df8b6724cdbfcdcfbb21d0b153c0 with result from task 43 parsl.dataflow.dflow DEBUG: Task 43 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 44 has memoization hash f96ed19427896a6032cb6dbc1b5e20b9 parsl.dataflow.memoization INFO: Task 44 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4bce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 44 try 0 launched on executor multi with executor id 823 parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cdc19b94-fd21-4f32-84fe-b3afc5905b7b_calibrate_1124707_20.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cdc19b94-fd21-4f32-84fe-b3afc5905b7b_calibrate_1124707_20.stderr parsl.dataflow.dflow INFO: Standard output for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8058a4c5-eee5-4ebb-9b01-a81dee60eb4c_characterizeImage_1124707_20.stdout parsl.dataflow.dflow INFO: Standard error for task 43 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8058a4c5-eee5-4ebb-9b01-a81dee60eb4c_characterizeImage_1124707_20.stderr parsl.dataflow.dflow INFO: Task 2 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b6993186717cf4791c4aa41deee6b3f8 with result from task 2 parsl.dataflow.dflow DEBUG: Task 2 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/38db5b39-d87d-4213-8686-09bce0132a2a_calibrate_1124831_45.stdout parsl.dataflow.dflow INFO: Standard error for task 2 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/38db5b39-d87d-4213-8686-09bce0132a2a_calibrate_1124831_45.stderr parsl.dataflow.dflow INFO: Task 356 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62cad87d931b9e1a023faf9c35445082 with result from task 356 parsl.dataflow.dflow DEBUG: Task 356 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/18892a54-5eb0-4555-a1a1-1b8d1420b5a6_calibrate_1124828_29.stdout parsl.dataflow.dflow INFO: Standard error for task 356 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/18892a54-5eb0-4555-a1a1-1b8d1420b5a6_calibrate_1124828_29.stderr parsl.dataflow.dflow INFO: Task 352 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d79cf44d11083d3a459e78f7e7992f2e with result from task 352 parsl.dataflow.dflow DEBUG: Task 352 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 353 has memoization hash e0cf6adb35172671d13f100e6c816cec parsl.dataflow.memoization INFO: Task 353 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8220> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 353 try 0 launched on executor multi with executor id 824 parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/423bf96a-6347-4665-902f-74613fd47d6b_calibrate_1124704_4.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/423bf96a-6347-4665-902f-74613fd47d6b_calibrate_1124704_4.stderr parsl.dataflow.dflow INFO: Standard output for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/efb151e0-9e3e-4ab4-a949-9c5c082e3998_characterizeImage_1124704_4.stdout parsl.dataflow.dflow INFO: Standard error for task 352 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/efb151e0-9e3e-4ab4-a949-9c5c082e3998_characterizeImage_1124704_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:15 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 5 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d55828ffa31ac2ac98223768907b90ec with result from task 5 parsl.dataflow.dflow DEBUG: Task 5 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/66e7bb53-41cd-44b8-ae66-1942aa2b568f_calibrate_1124831_3.stdout parsl.dataflow.dflow INFO: Standard error for task 5 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/66e7bb53-41cd-44b8-ae66-1942aa2b568f_calibrate_1124831_3.stderr parsl.dataflow.dflow INFO: Task 269 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2070557cf246ffc33f3b08b3c115102f with result from task 269 parsl.dataflow.dflow DEBUG: Task 269 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3fbcd8a9-b45c-4227-83a3-77c1b03d97de_calibrate_1124831_19.stdout parsl.dataflow.dflow INFO: Standard error for task 269 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3fbcd8a9-b45c-4227-83a3-77c1b03d97de_calibrate_1124831_19.stderr parsl.dataflow.dflow INFO: Task 805 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5d1ad48b119534cc3714d2f170368a52 with result from task 805 parsl.dataflow.dflow DEBUG: Task 805 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 806 has memoization hash 05974a8eee1b5254e9a8e0b98e51e2b9 parsl.dataflow.memoization INFO: Task 806 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cba60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 806 try 0 launched on executor multi with executor id 825 parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/c617d94a-0ac6-4b1f-ad1c-ce9f27e7ed95_calibrate_1124828_4.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/c617d94a-0ac6-4b1f-ad1c-ce9f27e7ed95_calibrate_1124828_4.stderr parsl.dataflow.dflow INFO: Standard output for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/76153e92-63c5-4522-81d7-3f9496d766fe_characterizeImage_1124828_4.stdout parsl.dataflow.dflow INFO: Standard error for task 805 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/76153e92-63c5-4522-81d7-3f9496d766fe_characterizeImage_1124828_4.stderr parsl.dataflow.dflow INFO: Task 520 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dc193fb503074e0b0b33f8b4336db520 with result from task 520 parsl.dataflow.dflow DEBUG: Task 520 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 521 has memoization hash 05ed2e741162230bb6d1a10d42056065 parsl.dataflow.memoization INFO: Task 521 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535298531a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 521 try 0 launched on executor multi with executor id 826 parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/d93da2a2-494f-42ed-b7fa-e142312b1f2c_calibrate_1124831_49.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/d93da2a2-494f-42ed-b7fa-e142312b1f2c_calibrate_1124831_49.stderr parsl.dataflow.dflow INFO: Standard output for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/90d12b2c-62d1-4afa-a459-e52083eafd74_characterizeImage_1124831_49.stdout parsl.dataflow.dflow INFO: Standard error for task 520 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/90d12b2c-62d1-4afa-a459-e52083eafd74_characterizeImage_1124831_49.stderr parsl.dataflow.dflow INFO: Task 688 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1b873a87a65fa96bf857407baa2f1e7 with result from task 688 parsl.dataflow.dflow DEBUG: Task 688 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 689 has memoization hash 0d6f9e2461620e8152fd09880d5c7f22 parsl.dataflow.memoization INFO: Task 689 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 689 try 0 launched on executor multi with executor id 827 parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ae522dc7-a361-46e7-b839-972d6a5f59c9_calibrate_1124704_21.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ae522dc7-a361-46e7-b839-972d6a5f59c9_calibrate_1124704_21.stderr parsl.dataflow.dflow INFO: Standard output for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/509c9765-30ac-420c-8f1d-fe050208357f_characterizeImage_1124704_21.stdout parsl.dataflow.dflow INFO: Standard error for task 688 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/509c9765-30ac-420c-8f1d-fe050208357f_characterizeImage_1124704_21.stderr parsl.dataflow.dflow INFO: Task 697 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 971ab3bce9fe6ad8fb9f2f0987caa812 with result from task 697 parsl.dataflow.dflow DEBUG: Task 697 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 698 has memoization hash fb4c321c8c9a21beb3e74d5aa7eafd79 parsl.dataflow.memoization INFO: Task 698 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973a8e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 698 try 0 launched on executor multi with executor id 828 parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/b2ac927f-5824-491c-92f6-9c1f827e008a_calibrate_1124710_36.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/b2ac927f-5824-491c-92f6-9c1f827e008a_calibrate_1124710_36.stderr parsl.dataflow.dflow INFO: Standard output for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d1431c49-73d9-41f4-aeea-961dbb8618cc_characterizeImage_1124710_36.stdout parsl.dataflow.dflow INFO: Standard error for task 697 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d1431c49-73d9-41f4-aeea-961dbb8618cc_characterizeImage_1124710_36.stderr parsl.dataflow.dflow INFO: Task 437 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9193f0f6dfe0c12d6f17af3140e7af98 with result from task 437 parsl.dataflow.dflow DEBUG: Task 437 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/fb8f1fc9-fa0c-4609-b6eb-0cfc90a8dddc_calibrate_1124831_44.stdout parsl.dataflow.dflow INFO: Standard error for task 437 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/fb8f1fc9-fa0c-4609-b6eb-0cfc90a8dddc_calibrate_1124831_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:16 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 199 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bf180cc6fa32d5269e7774965711c85f with result from task 199 parsl.dataflow.dflow DEBUG: Task 199 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 200 has memoization hash dfbad6d49ef53e715918ac468fd91f17 parsl.dataflow.memoization INFO: Task 200 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adae020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 200 try 0 launched on executor multi with executor id 829 parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/21cf64f8-5229-494f-8e69-40c05b93b165_calibrate_1124831_53.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/21cf64f8-5229-494f-8e69-40c05b93b165_calibrate_1124831_53.stderr parsl.dataflow.dflow INFO: Standard output for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/7e7d7f8a-9a40-4172-a067-f1b9071d79c9_characterizeImage_1124831_53.stdout parsl.dataflow.dflow INFO: Standard error for task 199 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/7e7d7f8a-9a40-4172-a067-f1b9071d79c9_characterizeImage_1124831_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:17 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 422 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d06068846ac7cc85b7f099daf34b8b8f with result from task 422 parsl.dataflow.dflow DEBUG: Task 422 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f56f3e96-f883-46e6-a55d-e4e175ea609c_calibrate_1124828_16.stdout parsl.dataflow.dflow INFO: Standard error for task 422 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f56f3e96-f883-46e6-a55d-e4e175ea609c_calibrate_1124828_16.stderr parsl.dataflow.dflow INFO: Task 838 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9acecf7e216a8b742aecf318d672ce76 with result from task 838 parsl.dataflow.dflow DEBUG: Task 838 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 839 has memoization hash b4bbbbc4f1c0465297e6278770f05bad parsl.dataflow.memoization INFO: Task 839 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cb7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 839 try 0 launched on executor multi with executor id 830 parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/045bb14c-af03-44a5-8888-c618d2cd0d7c_calibrate_1124710_14.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/045bb14c-af03-44a5-8888-c618d2cd0d7c_calibrate_1124710_14.stderr parsl.dataflow.dflow INFO: Standard output for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/4dfab188-f9c9-4443-a148-77afab903adc_characterizeImage_1124710_14.stdout parsl.dataflow.dflow INFO: Standard error for task 838 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/4dfab188-f9c9-4443-a148-77afab903adc_characterizeImage_1124710_14.stderr parsl.dataflow.dflow INFO: Task 292 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fa7cbf0f85b895f612f9413827df5257 with result from task 292 parsl.dataflow.dflow DEBUG: Task 292 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 293 has memoization hash ea06e7626690c34767c8a756a0a46824 parsl.dataflow.memoization INFO: Task 293 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cc180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 293 try 0 launched on executor multi with executor id 831 parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e50bd4df-9ec3-4921-8e00-e35e543f7089_calibrate_1124707_39.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e50bd4df-9ec3-4921-8e00-e35e543f7089_calibrate_1124707_39.stderr parsl.dataflow.dflow INFO: Standard output for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/f00b299f-8934-4119-b407-a12e9183a60a_characterizeImage_1124707_39.stdout parsl.dataflow.dflow INFO: Standard error for task 292 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/f00b299f-8934-4119-b407-a12e9183a60a_characterizeImage_1124707_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:18 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 572 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 362a914c492cd6e4cd9d0b24090fce01 with result from task 572 parsl.dataflow.dflow DEBUG: Task 572 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/ff773cea-520a-44b1-b70c-5addf5aba004_calibrate_1124831_22.stdout parsl.dataflow.dflow INFO: Standard error for task 572 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/ff773cea-520a-44b1-b70c-5addf5aba004_calibrate_1124831_22.stderr parsl.dataflow.dflow INFO: Task 374 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0180598a735c7f91db242f93fe811840 with result from task 374 parsl.dataflow.dflow DEBUG: Task 374 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/5e6dd009-02d0-4a75-a867-afcddf19492a_calibrate_1124831_29.stdout parsl.dataflow.dflow INFO: Standard error for task 374 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/5e6dd009-02d0-4a75-a867-afcddf19492a_calibrate_1124831_29.stderr parsl.dataflow.dflow INFO: Task 746 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 951234f2eeba18364b48f1bb7926268a with result from task 746 parsl.dataflow.dflow DEBUG: Task 746 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/779b319a-fcd0-4f6e-b949-09c464ab59a1_calibrate_1124831_39.stdout parsl.dataflow.dflow INFO: Standard error for task 746 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/779b319a-fcd0-4f6e-b949-09c464ab59a1_calibrate_1124831_39.stderr parsl.dataflow.dflow INFO: Task 175 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d848e99ab55ec2d9682d5e2a31ab68e with result from task 175 parsl.dataflow.dflow DEBUG: Task 175 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 176 has memoization hash 67043850a5958b0c50b31aaf1440eb6b parsl.dataflow.memoization INFO: Task 176 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adac720> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 176 try 0 launched on executor multi with executor id 832 parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/349f702f-1574-4dab-8f4e-55f4b921e9ed_calibrate_1124831_31.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/349f702f-1574-4dab-8f4e-55f4b921e9ed_calibrate_1124831_31.stderr parsl.dataflow.dflow INFO: Standard output for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/6c72ebea-cac5-4d8b-ba2d-84dba45c4f64_characterizeImage_1124831_31.stdout parsl.dataflow.dflow INFO: Standard error for task 175 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/6c72ebea-cac5-4d8b-ba2d-84dba45c4f64_characterizeImage_1124831_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:19 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 835 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 056474afebd676d12e87e5cb641cdd9e with result from task 835 parsl.dataflow.dflow DEBUG: Task 835 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 836 has memoization hash 5db2c4048fa4e0878bb5af706a5951ff parsl.dataflow.memoization INFO: Task 836 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cb100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 836 try 0 launched on executor multi with executor id 833 parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ced8c61e-52c4-456a-b7a3-a4a24c62ab7f_calibrate_1124707_57.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ced8c61e-52c4-456a-b7a3-a4a24c62ab7f_calibrate_1124707_57.stderr parsl.dataflow.dflow INFO: Standard output for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/647a9612-c987-48ed-9ae6-e5597189058d_characterizeImage_1124707_57.stdout parsl.dataflow.dflow INFO: Standard error for task 835 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/647a9612-c987-48ed-9ae6-e5597189058d_characterizeImage_1124707_57.stderr parsl.dataflow.dflow INFO: Task 41 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f482dd6223d62056751d72dc5198e8cd with result from task 41 parsl.dataflow.dflow DEBUG: Task 41 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/33b508ac-790f-4106-a9eb-b611305ad3c7_calibrate_1124828_32.stdout parsl.dataflow.dflow INFO: Standard error for task 41 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/33b508ac-790f-4106-a9eb-b611305ad3c7_calibrate_1124828_32.stderr parsl.dataflow.dflow INFO: Task 388 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52827528aa2758d1b71f76393e8e41b6 with result from task 388 parsl.dataflow.dflow DEBUG: Task 388 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 389 has memoization hash bd87eb0d204fa4b9f0179b02df9ae346 parsl.dataflow.memoization INFO: Task 389 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981dd00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 389 try 0 launched on executor multi with executor id 834 parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/babc7555-1937-46b1-8b76-20bfbb1417d3_calibrate_1124704_7.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/babc7555-1937-46b1-8b76-20bfbb1417d3_calibrate_1124704_7.stderr parsl.dataflow.dflow INFO: Standard output for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/641883f6-2955-4b61-b638-a6074541fa8f_characterizeImage_1124704_7.stdout parsl.dataflow.dflow INFO: Standard error for task 388 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/641883f6-2955-4b61-b638-a6074541fa8f_characterizeImage_1124704_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:20 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 329 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b2cfc7d394e521b2ade3c95a4141d866 with result from task 329 parsl.dataflow.dflow DEBUG: Task 329 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/25e47931-91d6-454c-b4a9-e563a3fb3dd9_calibrate_1124828_60.stdout parsl.dataflow.dflow INFO: Standard error for task 329 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/25e47931-91d6-454c-b4a9-e563a3fb3dd9_calibrate_1124828_60.stderr parsl.dataflow.dflow INFO: Task 790 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2ffeb30de1fc5dc813d52717a0430b1f with result from task 790 parsl.dataflow.dflow DEBUG: Task 790 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 791 has memoization hash 8443d45b5b83f27932bbc8b52e8ccfa6 parsl.dataflow.memoization INFO: Task 791 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c8c20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 791 try 0 launched on executor multi with executor id 835 parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/58000cf4-f5c1-4a2a-b9e5-75ae9042c03e_calibrate_1124704_47.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/58000cf4-f5c1-4a2a-b9e5-75ae9042c03e_calibrate_1124704_47.stderr parsl.dataflow.dflow INFO: Standard output for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b8daa3ec-07ba-44ac-9d00-2e5cbd0f7ac1_characterizeImage_1124704_47.stdout parsl.dataflow.dflow INFO: Standard error for task 790 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b8daa3ec-07ba-44ac-9d00-2e5cbd0f7ac1_characterizeImage_1124704_47.stderr parsl.dataflow.dflow INFO: Task 608 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 54ad86dd6135b5ef0387fbcfc0319f34 with result from task 608 parsl.dataflow.dflow DEBUG: Task 608 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/509c125f-07b9-45de-8498-6eeea71bf134_calibrate_1124828_1.stdout parsl.dataflow.dflow INFO: Standard error for task 608 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/509c125f-07b9-45de-8498-6eeea71bf134_calibrate_1124828_1.stderr parsl.dataflow.dflow INFO: Task 578 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 76c51aebd97b102067a1f06916e2b7e5 with result from task 578 parsl.dataflow.dflow DEBUG: Task 578 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/63ed195d-e62b-4530-976b-ef9470da7b03_calibrate_1124831_55.stdout parsl.dataflow.dflow INFO: Standard error for task 578 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/63ed195d-e62b-4530-976b-ef9470da7b03_calibrate_1124831_55.stderr parsl.dataflow.dflow INFO: Task 335 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09101877d7d3e8d8ef8d26d50c8c9bd0 with result from task 335 parsl.dataflow.dflow DEBUG: Task 335 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/cf367206-74fe-407f-8812-19ba9f6f38d7_calibrate_1124831_43.stdout parsl.dataflow.dflow INFO: Standard error for task 335 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/cf367206-74fe-407f-8812-19ba9f6f38d7_calibrate_1124831_43.stderr parsl.dataflow.dflow INFO: Task 301 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be866213b1313c6f8f9a1f6d5c559c1b with result from task 301 parsl.dataflow.dflow DEBUG: Task 301 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 302 has memoization hash 566ae98cdb1dcf16487d9abbd8177243 parsl.dataflow.memoization INFO: Task 302 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ccae0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 302 try 0 launched on executor multi with executor id 836 parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ad07d3a0-20a4-4495-9922-4d3978901d74_calibrate_1124707_43.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ad07d3a0-20a4-4495-9922-4d3978901d74_calibrate_1124707_43.stderr parsl.dataflow.dflow INFO: Standard output for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/9eb9a0e1-0ace-4ea9-97c4-17ef03e7dd89_characterizeImage_1124707_43.stdout parsl.dataflow.dflow INFO: Standard error for task 301 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/9eb9a0e1-0ace-4ea9-97c4-17ef03e7dd89_characterizeImage_1124707_43.stderr parsl.dataflow.dflow INFO: Task 524 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 103880317c3968ae686ad04ba44885b0 with result from task 524 parsl.dataflow.dflow DEBUG: Task 524 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/645d0830-5720-48c8-b9b2-df68b8df241a_calibrate_1124831_51.stdout parsl.dataflow.dflow INFO: Standard error for task 524 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/645d0830-5720-48c8-b9b2-df68b8df241a_calibrate_1124831_51.stderr parsl.dataflow.dflow INFO: Task 125 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0ce72d97fbbd3480112ce501d4f47fb8 with result from task 125 parsl.dataflow.dflow DEBUG: Task 125 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/7611d62a-1aff-4f2d-b994-f0f47abbf966_calibrate_1124831_28.stdout parsl.dataflow.dflow INFO: Standard error for task 125 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/7611d62a-1aff-4f2d-b994-f0f47abbf966_calibrate_1124831_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 265 parsl.dataflow.strategy DEBUG: Executor multi has 265 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:29:21 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local parsl.dataflow.dflow INFO: Task 307 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3d1f8f8f53032297b9fb9093e8646b4e with result from task 307 parsl.dataflow.dflow DEBUG: Task 307 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 308 has memoization hash 923e93ae896bee510feabeb2cbd94179 parsl.dataflow.memoization INFO: Task 308 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cd120> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 308 try 0 launched on executor multi with executor id 837 parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e0bf2b30-f9e1-4767-9d58-1459af999672_calibrate_1124707_60.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e0bf2b30-f9e1-4767-9d58-1459af999672_calibrate_1124707_60.stderr parsl.dataflow.dflow INFO: Standard output for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e8dd0d7f-f643-4245-8d9c-32cc3e1c9ce6_characterizeImage_1124707_60.stdout parsl.dataflow.dflow INFO: Standard error for task 307 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e8dd0d7f-f643-4245-8d9c-32cc3e1c9ce6_characterizeImage_1124707_60.stderr 2024-03-07 07:29:21 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 155 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fec8cf7665b15f340cff042e4d1c4ba with result from task 155 parsl.dataflow.dflow DEBUG: Task 155 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/21d3ec24-6717-4923-894a-ad03e174b095_calibrate_1124831_59.stdout parsl.dataflow.dflow INFO: Standard error for task 155 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/21d3ec24-6717-4923-894a-ad03e174b095_calibrate_1124831_59.stderr parsl.dataflow.dflow INFO: Task 19 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98af69f772a54546a5cedfad25cc06bf with result from task 19 parsl.dataflow.dflow DEBUG: Task 19 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 20 has memoization hash cc8ef5a44a42251ca0e13b07e76bbaff parsl.dataflow.memoization INFO: Task 20 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4a020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 20 try 0 launched on executor multi with executor id 838 parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/465da77e-a853-4003-8a44-8027012c10fc_calibrate_1124710_37.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/465da77e-a853-4003-8a44-8027012c10fc_calibrate_1124710_37.stderr parsl.dataflow.dflow INFO: Standard output for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/dcc67948-e9ee-468f-b6f1-31d90d0923e9_characterizeImage_1124710_37.stdout parsl.dataflow.dflow INFO: Standard error for task 19 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/dcc67948-e9ee-468f-b6f1-31d90d0923e9_characterizeImage_1124710_37.stderr parsl.dataflow.dflow INFO: Task 28 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b11caadc7d35a9e9b660e7862e7a5654 with result from task 28 parsl.dataflow.dflow DEBUG: Task 28 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 29 has memoization hash 0d0fb87c4d5ef8426ad43aedb68b9a28 parsl.dataflow.memoization INFO: Task 29 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4b740> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 29 try 0 launched on executor multi with executor id 839 parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/5614919d-83e8-4c19-8780-0e0bfe4c026b_calibrate_1124707_49.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/5614919d-83e8-4c19-8780-0e0bfe4c026b_calibrate_1124707_49.stderr parsl.dataflow.dflow INFO: Standard output for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e31696bb-a747-43b3-af42-a219d5e5abda_characterizeImage_1124707_49.stdout parsl.dataflow.dflow INFO: Standard error for task 28 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e31696bb-a747-43b3-af42-a219d5e5abda_characterizeImage_1124707_49.stderr parsl.dataflow.dflow INFO: Task 115 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 548c036581850f10fa601df41b430568 with result from task 115 parsl.dataflow.dflow DEBUG: Task 115 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 116 has memoization hash a1932d6d80f1a5deb0d4c2ab6cf432a1 parsl.dataflow.memoization INFO: Task 116 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8c860> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 116 try 0 launched on executor multi with executor id 840 parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/ca9316f3-2267-40e4-ba24-2b8fea2dc9d0_calibrate_1124828_58.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/ca9316f3-2267-40e4-ba24-2b8fea2dc9d0_calibrate_1124828_58.stderr parsl.dataflow.dflow INFO: Standard output for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/44a6bb3a-2a7f-4946-8b61-129c7e530646_characterizeImage_1124828_58.stdout parsl.dataflow.dflow INFO: Standard error for task 115 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/44a6bb3a-2a7f-4946-8b61-129c7e530646_characterizeImage_1124828_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:22 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 392 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83b64723061f95dba494bf25fe254bc8 with result from task 392 parsl.dataflow.dflow DEBUG: Task 392 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/48e16807-15c0-4d46-ae9b-a54336786d5d_calibrate_1124828_30.stdout parsl.dataflow.dflow INFO: Standard error for task 392 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/48e16807-15c0-4d46-ae9b-a54336786d5d_calibrate_1124828_30.stderr parsl.dataflow.dflow INFO: Task 602 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 209fefcca37df15a9e0fd224ae1eb26a with result from task 602 parsl.dataflow.dflow DEBUG: Task 602 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/37b62b35-237a-46e2-93b2-7c9269b1e41b_calibrate_1124828_10.stdout parsl.dataflow.dflow INFO: Standard error for task 602 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/37b62b35-237a-46e2-93b2-7c9269b1e41b_calibrate_1124828_10.stderr parsl.dataflow.dflow INFO: Task 629 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d434c693f1600aced80ba264f337f59 with result from task 629 parsl.dataflow.dflow DEBUG: Task 629 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/93d1524c-2d13-45e0-94d9-075be0d5c558_calibrate_1124831_35.stdout parsl.dataflow.dflow INFO: Standard error for task 629 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/93d1524c-2d13-45e0-94d9-075be0d5c558_calibrate_1124831_35.stderr parsl.dataflow.dflow INFO: Task 266 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ebe73d59e55599ce357174813aa0f4c8 with result from task 266 parsl.dataflow.dflow DEBUG: Task 266 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/2c6ccf6d-8ce1-4f7b-81f0-6292471b2bdc_calibrate_1124831_18.stdout parsl.dataflow.dflow INFO: Standard error for task 266 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/2c6ccf6d-8ce1-4f7b-81f0-6292471b2bdc_calibrate_1124831_18.stderr parsl.dataflow.dflow INFO: Task 446 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6a65a1cc8d1c6e0131308f001aea5290 with result from task 446 parsl.dataflow.dflow DEBUG: Task 446 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/1f83ff55-7c0f-4f6f-b7ab-2b24604863df_calibrate_1124828_45.stdout parsl.dataflow.dflow INFO: Standard error for task 446 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/1f83ff55-7c0f-4f6f-b7ab-2b24604863df_calibrate_1124828_45.stderr parsl.dataflow.dflow INFO: Task 928 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 011d3b2f31cc4ed785df6813668fcfc0 with result from task 928 parsl.dataflow.dflow DEBUG: Task 928 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 929 has memoization hash e4e81c19280a94b50f7c0ec4e4e7dbfb parsl.dataflow.memoization INFO: Task 929 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529665620> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 929 try 0 launched on executor multi with executor id 841 parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/4e0abbbc-d63e-47ad-b849-b3809b3e9ce9_calibrate_1124704_36.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/4e0abbbc-d63e-47ad-b849-b3809b3e9ce9_calibrate_1124704_36.stderr parsl.dataflow.dflow INFO: Standard output for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d23f19a8-2a87-444e-94f1-42ef0284233a_characterizeImage_1124704_36.stdout parsl.dataflow.dflow INFO: Standard error for task 928 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d23f19a8-2a87-444e-94f1-42ef0284233a_characterizeImage_1124704_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:23 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 685 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4345d15509a76aba6aec2b5e3942c009 with result from task 685 parsl.dataflow.dflow DEBUG: Task 685 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 686 has memoization hash cdf7fcf2652071fad348d995bebc35f0 parsl.dataflow.memoization INFO: Task 686 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529739d00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 686 try 0 launched on executor multi with executor id 842 parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/f55cbeed-aa88-43d6-b1dc-e6687d31e5fd_calibrate_1124707_4.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/f55cbeed-aa88-43d6-b1dc-e6687d31e5fd_calibrate_1124707_4.stderr parsl.dataflow.dflow INFO: Standard output for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/089dfc57-4ecd-4cc7-bf9f-c929da6d593a_characterizeImage_1124707_4.stdout parsl.dataflow.dflow INFO: Standard error for task 685 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/089dfc57-4ecd-4cc7-bf9f-c929da6d593a_characterizeImage_1124707_4.stderr parsl.dataflow.dflow INFO: Task 185 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 048ccb333097adaf1ff32d454173ebf3 with result from task 185 parsl.dataflow.dflow DEBUG: Task 185 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/4ee3d11a-9927-42d8-96fe-aa13d17dc53f_calibrate_1124828_41.stdout parsl.dataflow.dflow INFO: Standard error for task 185 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/4ee3d11a-9927-42d8-96fe-aa13d17dc53f_calibrate_1124828_41.stderr parsl.dataflow.dflow INFO: Task 820 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2dc3625834640182e3c2fc4ae98b5780 with result from task 820 parsl.dataflow.dflow DEBUG: Task 820 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 821 has memoization hash 55bf9172715b9c0eb0a8c420115edf56 parsl.dataflow.memoization INFO: Task 821 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295ca3e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 821 try 0 launched on executor multi with executor id 843 parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/fa205240-df9c-4c01-9105-d40fa1521ce7_calibrate_1124707_52.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/fa205240-df9c-4c01-9105-d40fa1521ce7_calibrate_1124707_52.stderr parsl.dataflow.dflow INFO: Standard output for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/24ffecc0-0d98-4a7b-b390-dfbf20733bd5_characterizeImage_1124707_52.stdout parsl.dataflow.dflow INFO: Standard error for task 820 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/24ffecc0-0d98-4a7b-b390-dfbf20733bd5_characterizeImage_1124707_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:24 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 677 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15389f9f4b1c4f77628210e38f5ab6e9 with result from task 677 parsl.dataflow.dflow DEBUG: Task 677 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/ee617dd9-b8e5-4929-9d4d-9a74e69301c9_calibrate_1124831_10.stdout parsl.dataflow.dflow INFO: Standard error for task 677 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/ee617dd9-b8e5-4929-9d4d-9a74e69301c9_calibrate_1124831_10.stderr parsl.dataflow.dflow INFO: Task 584 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50ff061f6e16ab19fe174432e90ac73b with result from task 584 parsl.dataflow.dflow DEBUG: Task 584 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/affd8d00-fe17-4aad-a01f-f52d446c5dc6_calibrate_1124828_54.stdout parsl.dataflow.dflow INFO: Standard error for task 584 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/affd8d00-fe17-4aad-a01f-f52d446c5dc6_calibrate_1124828_54.stderr parsl.dataflow.dflow INFO: Task 331 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cf1bec65d1b86f8e15519812ad7a4258 with result from task 331 parsl.dataflow.dflow DEBUG: Task 331 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 332 has memoization hash 94fae6f175c508f8a9acd5d06850b013 parsl.dataflow.memoization INFO: Task 332 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cea20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 332 try 0 launched on executor multi with executor id 844 parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/b032e008-4347-42e9-8477-d752c6a7e20b_calibrate_1124704_12.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/b032e008-4347-42e9-8477-d752c6a7e20b_calibrate_1124704_12.stderr parsl.dataflow.dflow INFO: Standard output for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f0cbeb6e-f4a7-439b-a200-b78899380d7b_characterizeImage_1124704_12.stdout parsl.dataflow.dflow INFO: Standard error for task 331 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f0cbeb6e-f4a7-439b-a200-b78899380d7b_characterizeImage_1124704_12.stderr parsl.dataflow.dflow INFO: Task 632 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 56609facf1705f845baf42ba34567468 with result from task 632 parsl.dataflow.dflow DEBUG: Task 632 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/29acc1eb-5e11-4835-b7ec-b2e1b61fd7f8_calibrate_1124828_14.stdout parsl.dataflow.dflow INFO: Standard error for task 632 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/29acc1eb-5e11-4835-b7ec-b2e1b61fd7f8_calibrate_1124828_14.stderr parsl.dataflow.dflow INFO: Task 50 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0f6af63142a1ae19ee57510b3de3b7b3 with result from task 50 parsl.dataflow.dflow DEBUG: Task 50 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/893d01de-ee1d-4bbf-9126-97ece8f1179d_calibrate_1124828_34.stdout parsl.dataflow.dflow INFO: Standard error for task 50 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/893d01de-ee1d-4bbf-9126-97ece8f1179d_calibrate_1124828_34.stderr parsl.dataflow.dflow INFO: Task 226 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 125201d7a8f0ff569a4b33bea3499481 with result from task 226 parsl.dataflow.dflow DEBUG: Task 226 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 227 has memoization hash 36e5337edac2418694c71a3abaf8f0bc parsl.dataflow.memoization INFO: Task 227 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adafc40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 227 try 0 launched on executor multi with executor id 845 parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/fbae5cf0-c419-49c2-92a1-022648a57d90_calibrate_1124707_37.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/fbae5cf0-c419-49c2-92a1-022648a57d90_calibrate_1124707_37.stderr parsl.dataflow.dflow INFO: Standard output for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/00c99fee-79fe-4691-a5a0-a6ca0b62e39b_characterizeImage_1124707_37.stdout parsl.dataflow.dflow INFO: Standard error for task 226 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/00c99fee-79fe-4691-a5a0-a6ca0b62e39b_characterizeImage_1124707_37.stderr parsl.dataflow.dflow INFO: Task 166 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee88682a25caf486b8fc1458b2dff22d with result from task 166 parsl.dataflow.dflow DEBUG: Task 166 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 167 has memoization hash 7bac7785d0c32a44c5424d373ddd0930 parsl.dataflow.memoization INFO: Task 167 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8fd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 167 try 0 launched on executor multi with executor id 846 parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/72a47e40-27e5-436c-8c84-99e14e8c5b38_calibrate_1124704_6.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/72a47e40-27e5-436c-8c84-99e14e8c5b38_calibrate_1124704_6.stderr parsl.dataflow.dflow INFO: Standard output for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7912c118-5d91-4783-b108-13fe11b4eaec_characterizeImage_1124704_6.stdout parsl.dataflow.dflow INFO: Standard error for task 166 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7912c118-5d91-4783-b108-13fe11b4eaec_characterizeImage_1124704_6.stderr parsl.dataflow.dflow INFO: Task 74 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b8b4c1a80916d57112d59d09af1dcd4c with result from task 74 parsl.dataflow.dflow DEBUG: Task 74 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/c7079ebb-db94-415d-93ee-17fb488ee03c_calibrate_1124831_58.stdout parsl.dataflow.dflow INFO: Standard error for task 74 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/c7079ebb-db94-415d-93ee-17fb488ee03c_calibrate_1124831_58.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:25 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 494 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 162d0c20ce23573ef27af89b582af006 with result from task 494 parsl.dataflow.dflow DEBUG: Task 494 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/99ae0455-9089-4e02-8ee3-3f6ae1a88997_calibrate_1124828_20.stdout parsl.dataflow.dflow INFO: Standard error for task 494 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/99ae0455-9089-4e02-8ee3-3f6ae1a88997_calibrate_1124828_20.stderr parsl.dataflow.dflow INFO: Task 434 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1a08ee0edf32928167d71f3464fe4e6e with result from task 434 parsl.dataflow.dflow DEBUG: Task 434 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/ddea7783-92f1-47bf-8e7e-98872f29e5e1_calibrate_1124828_36.stdout parsl.dataflow.dflow INFO: Standard error for task 434 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/ddea7783-92f1-47bf-8e7e-98872f29e5e1_calibrate_1124828_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:26 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 398 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c3781f310ec439a94b6c783bac56999f with result from task 398 parsl.dataflow.dflow DEBUG: Task 398 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/d5447731-dd06-4e1b-8227-d8fa5a6286e5_calibrate_1124828_46.stdout parsl.dataflow.dflow INFO: Standard error for task 398 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/d5447731-dd06-4e1b-8227-d8fa5a6286e5_calibrate_1124828_46.stderr parsl.dataflow.dflow INFO: Task 530 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbf8f14396eaa375a9473bfcaca51bdf with result from task 530 parsl.dataflow.dflow DEBUG: Task 530 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a9adc866-6156-4a91-a82c-c1f83ae0316b_calibrate_1124828_50.stdout parsl.dataflow.dflow INFO: Standard error for task 530 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a9adc866-6156-4a91-a82c-c1f83ae0316b_calibrate_1124828_50.stderr parsl.dataflow.dflow INFO: Task 796 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 807d63d0413ae0202eb0b3ad41ae0f70 with result from task 796 parsl.dataflow.dflow DEBUG: Task 796 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 797 has memoization hash 4d3c800a346dcfcc4f2d0d000980c086 parsl.dataflow.memoization INFO: Task 797 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 797 try 0 launched on executor multi with executor id 847 parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/927f6d5e-5aad-48c8-bc6a-16e772cc1e22_calibrate_1124710_13.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/927f6d5e-5aad-48c8-bc6a-16e772cc1e22_calibrate_1124710_13.stderr parsl.dataflow.dflow INFO: Standard output for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f29dedc8-699e-4eca-a2e2-95164dcbe728_characterizeImage_1124710_13.stdout parsl.dataflow.dflow INFO: Standard error for task 796 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/f29dedc8-699e-4eca-a2e2-95164dcbe728_characterizeImage_1124710_13.stderr parsl.dataflow.dflow INFO: Task 586 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92e5fee8042d7483ae63c3e0a5f2ffb7 with result from task 586 parsl.dataflow.dflow DEBUG: Task 586 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 587 has memoization hash 98b8dde43cb7fb4a2c2551479652ae7c parsl.dataflow.memoization INFO: Task 587 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 587 try 0 launched on executor multi with executor id 848 parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/a4b922bc-feab-4e36-9698-5d01c542b3ad_calibrate_1124707_21.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/a4b922bc-feab-4e36-9698-5d01c542b3ad_calibrate_1124707_21.stderr parsl.dataflow.dflow INFO: Standard output for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/c2d56486-e16a-4f8d-9186-2fb8c2115d91_characterizeImage_1124707_21.stdout parsl.dataflow.dflow INFO: Standard error for task 586 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/c2d56486-e16a-4f8d-9186-2fb8c2115d91_characterizeImage_1124707_21.stderr parsl.dataflow.dflow INFO: Task 407 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5f3438cfd9d9d9885707b7b3f085ef65 with result from task 407 parsl.dataflow.dflow DEBUG: Task 407 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/62b1a9b4-29e9-4066-aa3a-c968e27f2813_calibrate_1124828_37.stdout parsl.dataflow.dflow INFO: Standard error for task 407 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/62b1a9b4-29e9-4066-aa3a-c968e27f2813_calibrate_1124828_37.stderr parsl.dataflow.dflow INFO: Task 841 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84050f9a282564ff9a17ec3624b94f74 with result from task 841 parsl.dataflow.dflow DEBUG: Task 841 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 842 has memoization hash c463e4e15015838451350448e78d4e01 parsl.dataflow.memoization INFO: Task 842 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529608180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 842 try 0 launched on executor multi with executor id 849 parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/240aaa11-7a31-43f2-80bb-230ad0deae0c_calibrate_1124828_39.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/240aaa11-7a31-43f2-80bb-230ad0deae0c_calibrate_1124828_39.stderr parsl.dataflow.dflow INFO: Standard output for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/6531013e-4d6c-4bc2-95e1-e61b9e65aaf8_characterizeImage_1124828_39.stdout parsl.dataflow.dflow INFO: Standard error for task 841 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/6531013e-4d6c-4bc2-95e1-e61b9e65aaf8_characterizeImage_1124828_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:27 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 539 try 0 failed parsl.dataflow.dflow INFO: Task 539 marked for retry parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/83c52c40-c27e-433c-98bf-f968d4880528_calibrate_1124707_31.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/83c52c40-c27e-433c-98bf-f968d4880528_calibrate_1124707_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 539 has memoization hash e171923504aa2c4ccbb268b2f6830fc6 parsl.dataflow.memoization INFO: Task 539 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969e2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 539 try 1 launched on executor multi with executor id 850 parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/83c52c40-c27e-433c-98bf-f968d4880528_calibrate_1124707_31.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/83c52c40-c27e-433c-98bf-f968d4880528_calibrate_1124707_31.stderr parsl.dataflow.dflow INFO: Task 26 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c92466e146db939895edbef77e14f34f with result from task 26 parsl.dataflow.dflow DEBUG: Task 26 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/cde1c650-0607-4ed6-a84b-14071e7ba660_calibrate_1124828_51.stdout parsl.dataflow.dflow INFO: Standard error for task 26 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/cde1c650-0607-4ed6-a84b-14071e7ba660_calibrate_1124828_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:28 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 766 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 321a37dc4ea997c54169ac4568ee2a17 with result from task 766 parsl.dataflow.dflow DEBUG: Task 766 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash c715fbad0dfe0df5530a38b18d4a0e92 parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 0 launched on executor multi with executor id 851 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/46d083a0-25ac-438f-ac7e-fa79f51f9dd8_calibrate_1124704_31.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/46d083a0-25ac-438f-ac7e-fa79f51f9dd8_calibrate_1124704_31.stderr parsl.dataflow.dflow INFO: Standard output for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/1da3ea7d-9792-4592-9dcf-2b2810d8639d_characterizeImage_1124704_31.stdout parsl.dataflow.dflow INFO: Standard error for task 766 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/1da3ea7d-9792-4592-9dcf-2b2810d8639d_characterizeImage_1124704_31.stderr parsl.dataflow.dflow INFO: Task 34 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f72090c0fd13cf76aceb002d59222c6d with result from task 34 parsl.dataflow.dflow DEBUG: Task 34 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 35 has memoization hash 295d8e24c58446b4cb86e115c6933678 parsl.dataflow.memoization INFO: Task 35 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4b100> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 35 try 0 launched on executor multi with executor id 852 parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/080030d8-7ae4-45a9-a4bc-8e007a14aa64_calibrate_1124710_25.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/080030d8-7ae4-45a9-a4bc-8e007a14aa64_calibrate_1124710_25.stderr parsl.dataflow.dflow INFO: Standard output for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/01ad5621-feac-4d5d-ba7b-d60a2d17e021_characterizeImage_1124710_25.stdout parsl.dataflow.dflow INFO: Standard error for task 34 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/01ad5621-feac-4d5d-ba7b-d60a2d17e021_characterizeImage_1124710_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:29 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 340 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e9dbbfab1456e45e952bda3ce2a8b6aa with result from task 340 parsl.dataflow.dflow DEBUG: Task 340 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 341 has memoization hash d5994f0617d45d5b0635f28f3be96652 parsl.dataflow.memoization INFO: Task 341 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cf380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 341 try 0 launched on executor multi with executor id 853 parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ef9c1552-afaa-4c22-8b3f-45ae2fd52883_calibrate_1124707_34.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ef9c1552-afaa-4c22-8b3f-45ae2fd52883_calibrate_1124707_34.stderr parsl.dataflow.dflow INFO: Standard output for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/653d08f7-25f6-4f36-90c6-84f4f4a0ac90_characterizeImage_1124707_34.stdout parsl.dataflow.dflow INFO: Standard error for task 340 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/653d08f7-25f6-4f36-90c6-84f4f4a0ac90_characterizeImage_1124707_34.stderr parsl.dataflow.dflow INFO: Task 235 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01be0a2bdc1fd9724d2f6666bf9ca693 with result from task 235 parsl.dataflow.dflow DEBUG: Task 235 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 236 has memoization hash 33dc930996343adde85bb4ca6515b3d5 parsl.dataflow.memoization INFO: Task 236 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a85e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 236 try 0 launched on executor multi with executor id 854 parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8a1a315a-4e76-4309-9090-a49e2a13161e_calibrate_1124704_43.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8a1a315a-4e76-4309-9090-a49e2a13161e_calibrate_1124704_43.stderr parsl.dataflow.dflow INFO: Standard output for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/462d258e-d17d-454d-a9f1-58acac371f50_characterizeImage_1124704_43.stdout parsl.dataflow.dflow INFO: Standard error for task 235 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/462d258e-d17d-454d-a9f1-58acac371f50_characterizeImage_1124704_43.stderr parsl.dataflow.dflow INFO: Task 620 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9730da99ee13aee5afd2aeb7475936a7 with result from task 620 parsl.dataflow.dflow DEBUG: Task 620 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/cefaf84b-d60e-4ca0-8c17-4ff606a213b2_calibrate_1124828_55.stdout parsl.dataflow.dflow INFO: Standard error for task 620 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/cefaf84b-d60e-4ca0-8c17-4ff606a213b2_calibrate_1124828_55.stderr parsl.dataflow.dflow INFO: Task 92 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 93f1d6d780912aedb08575c426ab908a with result from task 92 parsl.dataflow.dflow DEBUG: Task 92 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/49c4b003-32aa-4ce8-9605-10d69731c5e9_calibrate_1124831_25.stdout parsl.dataflow.dflow INFO: Standard error for task 92 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/49c4b003-32aa-4ce8-9605-10d69731c5e9_calibrate_1124831_25.stderr parsl.dataflow.dflow INFO: Task 731 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 88111540117c35374158c7bcc476f857 with result from task 731 parsl.dataflow.dflow DEBUG: Task 731 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/6bc76c9e-1f3f-4ad8-9697-70b83104d397_calibrate_1124831_62.stdout parsl.dataflow.dflow INFO: Standard error for task 731 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/6bc76c9e-1f3f-4ad8-9697-70b83104d397_calibrate_1124831_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:30 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 916 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 18120ba72e0bf9793414613fd666429b with result from task 916 parsl.dataflow.dflow DEBUG: Task 916 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 917 has memoization hash ba9b562ef4b620eef502338a76eff815 parsl.dataflow.memoization INFO: Task 917 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296649a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 917 try 0 launched on executor multi with executor id 855 parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/da210bcc-6a21-4771-adf1-348658d70a6a_calibrate_1124828_24.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/da210bcc-6a21-4771-adf1-348658d70a6a_calibrate_1124828_24.stderr parsl.dataflow.dflow INFO: Standard output for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/67676cf3-fd81-462b-97bc-07793e84167a_characterizeImage_1124828_24.stdout parsl.dataflow.dflow INFO: Standard error for task 916 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/67676cf3-fd81-462b-97bc-07793e84167a_characterizeImage_1124828_24.stderr parsl.dataflow.dflow INFO: Task 394 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29f1f247b39ded91c94eb3e58ecf1b54 with result from task 394 parsl.dataflow.dflow DEBUG: Task 394 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 395 has memoization hash e2b2ddc3c4fee455d986bbd3586a07a8 parsl.dataflow.memoization INFO: Task 395 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981d800> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 395 try 0 launched on executor multi with executor id 856 parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ef59c1e1-6726-41f8-9d87-42fa659ebcbc_calibrate_1124710_48.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ef59c1e1-6726-41f8-9d87-42fa659ebcbc_calibrate_1124710_48.stderr parsl.dataflow.dflow INFO: Standard output for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7fa8bf0e-1d8a-4caf-8b82-3cc9377a91f7_characterizeImage_1124710_48.stdout parsl.dataflow.dflow INFO: Standard error for task 394 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7fa8bf0e-1d8a-4caf-8b82-3cc9377a91f7_characterizeImage_1124710_48.stderr parsl.dataflow.dflow INFO: Task 299 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42f8628fa8fe6659db0243e7cdd49829 with result from task 299 parsl.dataflow.dflow DEBUG: Task 299 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8a9c78e9-7bde-4af8-a76e-9be36c7d6b65_calibrate_1124828_43.stdout parsl.dataflow.dflow INFO: Standard error for task 299 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8a9c78e9-7bde-4af8-a76e-9be36c7d6b65_calibrate_1124828_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:31 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 752 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 11656a062f8ade835d92f43fc67cee15 with result from task 752 parsl.dataflow.dflow DEBUG: Task 752 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a2c6538f-badd-4370-9599-f325a65c3818_calibrate_1124831_11.stdout parsl.dataflow.dflow INFO: Standard error for task 752 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a2c6538f-badd-4370-9599-f325a65c3818_calibrate_1124831_11.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 107 try 0 failed parsl.dataflow.dflow INFO: Task 107 marked for retry parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/526ba917-d599-4483-8704-f932cf8e8631_calibrate_1124710_31.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/526ba917-d599-4483-8704-f932cf8e8631_calibrate_1124710_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 107 has memoization hash bcd408b2aee6cf23dfa118d4f35ab8cd parsl.dataflow.memoization INFO: Task 107 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad67e20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 107 try 1 launched on executor multi with executor id 857 parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/526ba917-d599-4483-8704-f932cf8e8631_calibrate_1124710_31.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/526ba917-d599-4483-8704-f932cf8e8631_calibrate_1124710_31.stderr parsl.dataflow.dflow INFO: Task 460 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f24ac40c8e757bdfebb81bdca0130d5f with result from task 460 parsl.dataflow.dflow DEBUG: Task 460 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 461 has memoization hash 50e1e8ccb74eb426f0478d17e95b4d18 parsl.dataflow.memoization INFO: Task 461 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 461 try 0 launched on executor multi with executor id 858 parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/983f364a-1011-4e6e-91e1-67fd8f70e2be_calibrate_1124710_52.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/983f364a-1011-4e6e-91e1-67fd8f70e2be_calibrate_1124710_52.stderr parsl.dataflow.dflow INFO: Standard output for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/188137e2-e3bc-4d8e-a172-e2d2e6212b31_characterizeImage_1124710_52.stdout parsl.dataflow.dflow INFO: Standard error for task 460 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/188137e2-e3bc-4d8e-a172-e2d2e6212b31_characterizeImage_1124710_52.stderr parsl.dataflow.dflow INFO: Task 692 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5451db0da0292acd02401f200987ea74 with result from task 692 parsl.dataflow.dflow DEBUG: Task 692 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/d78867cc-bea0-43c8-84da-8840712ba6e3_calibrate_1124828_53.stdout parsl.dataflow.dflow INFO: Standard error for task 692 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/d78867cc-bea0-43c8-84da-8840712ba6e3_calibrate_1124828_53.stderr parsl.dataflow.dflow INFO: Task 862 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ee6d44b90bd39946586a0b4262e2b2ea with result from task 862 parsl.dataflow.dflow DEBUG: Task 862 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 863 has memoization hash 691031f50695038f236a02c12a2dc80f parsl.dataflow.memoization INFO: Task 863 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529609940> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 863 try 0 launched on executor multi with executor id 859 parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/b093d3e4-fd60-4b76-944a-35b0fffb943e_calibrate_1124707_28.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/b093d3e4-fd60-4b76-944a-35b0fffb943e_calibrate_1124707_28.stderr parsl.dataflow.dflow INFO: Standard output for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/a9dc3d83-6d39-43bd-b9cd-1c69200cefbe_characterizeImage_1124707_28.stdout parsl.dataflow.dflow INFO: Standard error for task 862 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/a9dc3d83-6d39-43bd-b9cd-1c69200cefbe_characterizeImage_1124707_28.stderr parsl.dataflow.dflow INFO: Task 281 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7e26d30eee56f747004ee4c85b9f4d0 with result from task 281 parsl.dataflow.dflow DEBUG: Task 281 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4ab71671-00e2-4a47-82cd-fe15ff7cb093_calibrate_1124831_52.stdout parsl.dataflow.dflow INFO: Standard error for task 281 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4ab71671-00e2-4a47-82cd-fe15ff7cb093_calibrate_1124831_52.stderr parsl.dataflow.dflow INFO: Task 650 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b9748bbca32b585bbdbe7dd58ca09e20 with result from task 650 parsl.dataflow.dflow DEBUG: Task 650 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a2220a61-1200-4532-8a83-d6260d199684_calibrate_1124831_7.stdout parsl.dataflow.dflow INFO: Standard error for task 650 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a2220a61-1200-4532-8a83-d6260d199684_calibrate_1124831_7.stderr parsl.dataflow.dflow INFO: Task 37 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry abf1dc4e977150ff1ec666dd6469c200 with result from task 37 parsl.dataflow.dflow DEBUG: Task 37 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 38 has memoization hash 4f5b8372c818f4ac9476aa52f7b98485 parsl.dataflow.memoization INFO: Task 38 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad4ab60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 38 try 0 launched on executor multi with executor id 860 parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/2ea5cb51-2149-49c6-8189-52fcbaa11c74_calibrate_1124707_25.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/2ea5cb51-2149-49c6-8189-52fcbaa11c74_calibrate_1124707_25.stderr parsl.dataflow.dflow INFO: Standard output for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e0a3a9c0-179c-4c7c-8e59-d3925bb6057b_characterizeImage_1124707_25.stdout parsl.dataflow.dflow INFO: Standard error for task 37 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e0a3a9c0-179c-4c7c-8e59-d3925bb6057b_characterizeImage_1124707_25.stderr parsl.dataflow.dflow INFO: Task 803 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 62a816b66796c91209939bcd50c11707 with result from task 803 parsl.dataflow.dflow DEBUG: Task 803 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3aabd228-c636-4d20-8b37-9021c6e4df8d_calibrate_1124831_41.stdout parsl.dataflow.dflow INFO: Standard error for task 803 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3aabd228-c636-4d20-8b37-9021c6e4df8d_calibrate_1124831_41.stderr parsl.dataflow.dflow INFO: Task 484 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8d062f724d22b1973c557d55c9bb038a with result from task 484 parsl.dataflow.dflow DEBUG: Task 484 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 485 has memoization hash fc96c72b7eb2d17d9a0e3b9268169fb0 parsl.dataflow.memoization INFO: Task 485 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529850540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 485 try 0 launched on executor multi with executor id 861 parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/95b85a38-ed3a-4871-8ba9-bac964c14f8b_calibrate_1124710_34.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/95b85a38-ed3a-4871-8ba9-bac964c14f8b_calibrate_1124710_34.stderr parsl.dataflow.dflow INFO: Standard output for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7006ff7b-5f8a-4b99-b95a-9d4c9ed70c83_characterizeImage_1124710_34.stdout parsl.dataflow.dflow INFO: Standard error for task 484 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7006ff7b-5f8a-4b99-b95a-9d4c9ed70c83_characterizeImage_1124710_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:32 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 500 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aee7c85a6b9736e255774f553986d9c3 with result from task 500 parsl.dataflow.dflow DEBUG: Task 500 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4f20c361-b818-428b-ba02-26f82a5d742d_calibrate_1124831_13.stdout parsl.dataflow.dflow INFO: Standard error for task 500 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4f20c361-b818-428b-ba02-26f82a5d742d_calibrate_1124831_13.stderr parsl.dataflow.dflow INFO: Task 23 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bcde225ec57bd5f3865a97b82032df08 with result from task 23 parsl.dataflow.dflow DEBUG: Task 23 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f5c4a657-79f1-4503-abd4-51205663bb7b_calibrate_1124828_21.stdout parsl.dataflow.dflow INFO: Standard error for task 23 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f5c4a657-79f1-4503-abd4-51205663bb7b_calibrate_1124828_21.stderr parsl.dataflow.dflow INFO: Task 244 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38ad56c0641f9c747024ac2cc09a738b with result from task 244 parsl.dataflow.dflow DEBUG: Task 244 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 245 has memoization hash 6ca111a9f953d7cd6ed4f9b6209d98a1 parsl.dataflow.memoization INFO: Task 245 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a8f40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 245 try 0 launched on executor multi with executor id 862 parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/5c44a510-a23e-479e-a3b0-d327b0af5722_calibrate_1124704_27.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/5c44a510-a23e-479e-a3b0-d327b0af5722_calibrate_1124704_27.stderr parsl.dataflow.dflow INFO: Standard output for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/db1da635-c7b6-464c-98b2-1754d7da7e5e_characterizeImage_1124704_27.stdout parsl.dataflow.dflow INFO: Standard error for task 244 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/db1da635-c7b6-464c-98b2-1754d7da7e5e_characterizeImage_1124704_27.stderr parsl.dataflow.dflow INFO: Task 85 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4e0801ffd36ecfa1ab29fef91e08616 with result from task 85 parsl.dataflow.dflow DEBUG: Task 85 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 86 has memoization hash 0505f7930b14683c69a0220b00990e51 parsl.dataflow.memoization INFO: Task 86 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad668e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 86 try 0 launched on executor multi with executor id 863 parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7000406f-7cc7-4ca8-866c-560e312af954_calibrate_1124710_55.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7000406f-7cc7-4ca8-866c-560e312af954_calibrate_1124710_55.stderr parsl.dataflow.dflow INFO: Standard output for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/611dc2bd-957b-4f10-9cf9-49b0d9f3b2c8_characterizeImage_1124710_55.stdout parsl.dataflow.dflow INFO: Standard error for task 85 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/611dc2bd-957b-4f10-9cf9-49b0d9f3b2c8_characterizeImage_1124710_55.stderr parsl.dataflow.dflow INFO: Task 809 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c6eacf41c434061aaca70ddd1f2b796a with result from task 809 parsl.dataflow.dflow DEBUG: Task 809 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a75da45d-62c3-49a4-be30-c4568d89da20_calibrate_1124828_57.stdout parsl.dataflow.dflow INFO: Standard error for task 809 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a75da45d-62c3-49a4-be30-c4568d89da20_calibrate_1124828_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:33 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 832 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4e55c91c43ca41ddcc1b3d93dc20bae2 with result from task 832 parsl.dataflow.dflow DEBUG: Task 832 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 833 has memoization hash b5d8ce2d92e392c916c4aa33e4d240a9 parsl.dataflow.memoization INFO: Task 833 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cb060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 833 try 0 launched on executor multi with executor id 864 parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ec455e3f-4689-4b40-9f42-3f1a3d648a2e_calibrate_1124704_13.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ec455e3f-4689-4b40-9f42-3f1a3d648a2e_calibrate_1124704_13.stderr parsl.dataflow.dflow INFO: Standard output for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/4351cc4b-11d2-458b-a7c8-e6484c15b62d_characterizeImage_1124704_13.stdout parsl.dataflow.dflow INFO: Standard error for task 832 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/4351cc4b-11d2-458b-a7c8-e6484c15b62d_characterizeImage_1124704_13.stderr parsl.dataflow.dflow INFO: Task 851 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 059ed7e15fd665eee53c902d5affdab2 with result from task 851 parsl.dataflow.dflow DEBUG: Task 851 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/da30db92-b6d9-465c-9cb5-f94498025fb5_calibrate_1124828_8.stdout parsl.dataflow.dflow INFO: Standard error for task 851 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/da30db92-b6d9-465c-9cb5-f94498025fb5_calibrate_1124828_8.stderr parsl.dataflow.dflow INFO: Task 922 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d8b8c71f3489fe39de044a9c44e5974 with result from task 922 parsl.dataflow.dflow DEBUG: Task 922 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 923 has memoization hash eb74c148f5615fbada1e80da85486b80 parsl.dataflow.memoization INFO: Task 923 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529665080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 923 try 0 launched on executor multi with executor id 865 parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/5e53fbdc-d13a-4d27-b244-bab6e7f8d125_calibrate_1124831_48.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/5e53fbdc-d13a-4d27-b244-bab6e7f8d125_calibrate_1124831_48.stderr parsl.dataflow.dflow INFO: Standard output for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/2187b440-49a9-446e-9ff8-ea79727cec2b_characterizeImage_1124831_48.stdout parsl.dataflow.dflow INFO: Standard error for task 922 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/2187b440-49a9-446e-9ff8-ea79727cec2b_characterizeImage_1124831_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:34 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 613 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5818de60ea699db1e404da198f958abd with result from task 613 parsl.dataflow.dflow DEBUG: Task 613 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 614 has memoization hash a7a07175145d1a60fd321d3679042495 parsl.dataflow.memoization INFO: Task 614 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296e8d60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 614 try 0 launched on executor multi with executor id 866 parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/3a9171dd-200e-4199-b878-cefa50df8ff3_calibrate_1124710_24.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/3a9171dd-200e-4199-b878-cefa50df8ff3_calibrate_1124710_24.stderr parsl.dataflow.dflow INFO: Standard output for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/2af2eed8-323c-489d-ba5c-36a56e3423ee_characterizeImage_1124710_24.stdout parsl.dataflow.dflow INFO: Standard error for task 613 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/2af2eed8-323c-489d-ba5c-36a56e3423ee_characterizeImage_1124710_24.stderr parsl.dataflow.dflow INFO: Task 812 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4bae2d6e143a8155c822987bd110194 with result from task 812 parsl.dataflow.dflow DEBUG: Task 812 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/79e4cf3f-dfdb-4eed-8a62-e6e9ef6d70ee_calibrate_1124828_6.stdout parsl.dataflow.dflow INFO: Standard error for task 812 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/79e4cf3f-dfdb-4eed-8a62-e6e9ef6d70ee_calibrate_1124828_6.stderr parsl.dataflow.dflow INFO: Task 566 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a2c0069405c97118d2d7a972cd23c9d7 with result from task 566 parsl.dataflow.dflow DEBUG: Task 566 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/3dc83a89-f9d5-4637-b270-4bfda7253cd2_calibrate_1124828_33.stdout parsl.dataflow.dflow INFO: Standard error for task 566 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/3dc83a89-f9d5-4637-b270-4bfda7253cd2_calibrate_1124828_33.stderr parsl.dataflow.dflow INFO: Task 818 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry afdd9b0cd317dea1760bd8ad8d26f0d1 with result from task 818 parsl.dataflow.dflow DEBUG: Task 818 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/8c7811af-2c78-4e71-a9a2-844b093f5a93_calibrate_1124831_17.stdout parsl.dataflow.dflow INFO: Standard error for task 818 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/8c7811af-2c78-4e71-a9a2-844b093f5a93_calibrate_1124831_17.stderr parsl.dataflow.dflow INFO: Task 826 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f8e345a93057fcae6008c8e2461fa8c0 with result from task 826 parsl.dataflow.dflow DEBUG: Task 826 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 827 has memoization hash 0408185fd595f65ab2f6dee326701fac parsl.dataflow.memoization INFO: Task 827 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295caa20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 827 try 0 launched on executor multi with executor id 867 parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/de8d0c87-543a-4416-a655-76e5f4208abb_calibrate_1124704_32.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/de8d0c87-543a-4416-a655-76e5f4208abb_calibrate_1124704_32.stderr parsl.dataflow.dflow INFO: Standard output for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f92cb868-7e4b-4405-946b-48e6d8c5edf5_characterizeImage_1124704_32.stdout parsl.dataflow.dflow INFO: Standard error for task 826 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f92cb868-7e4b-4405-946b-48e6d8c5edf5_characterizeImage_1124704_32.stderr parsl.dataflow.dflow INFO: Task 289 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3788ce2c96b8e2917c24e3dd54720071 with result from task 289 parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:35 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow DEBUG: Task 289 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 290 has memoization hash 479715092e37b2986eb405639cfc3b7b parsl.dataflow.memoization INFO: Task 290 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297abf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 290 try 0 launched on executor multi with executor id 868 parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/0f87c0c3-ae36-41c9-8937-9d46fd4badb9_calibrate_1124710_28.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/0f87c0c3-ae36-41c9-8937-9d46fd4badb9_calibrate_1124710_28.stderr parsl.dataflow.dflow INFO: Standard output for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/8cd38eda-2837-4f90-9dfa-d15247aa2c17_characterizeImage_1124710_28.stdout parsl.dataflow.dflow INFO: Standard error for task 289 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/8cd38eda-2837-4f90-9dfa-d15247aa2c17_characterizeImage_1124710_28.stderr parsl.dataflow.dflow INFO: Task 188 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5e07233474a6068e1cdeb596c35c8566 with result from task 188 parsl.dataflow.dflow DEBUG: Task 188 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/5173662b-31b4-4b73-bcf5-34f2943d5b60_calibrate_1124828_42.stdout parsl.dataflow.dflow INFO: Standard error for task 188 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/5173662b-31b4-4b73-bcf5-34f2943d5b60_calibrate_1124828_42.stderr parsl.dataflow.dflow INFO: Task 517 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09dc2e7e8196d00430d9c9aaa697d052 with result from task 517 parsl.dataflow.dflow DEBUG: Task 517 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 518 has memoization hash 9ef063c79aac55effdd6b3fa8e5cad70 parsl.dataflow.memoization INFO: Task 518 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529852f20> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 518 try 0 launched on executor multi with executor id 869 parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a484d460-9410-43f5-b221-d39ec845628a_calibrate_1124831_24.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a484d460-9410-43f5-b221-d39ec845628a_calibrate_1124831_24.stderr parsl.dataflow.dflow INFO: Standard output for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/3af21e47-c9d6-470d-bac3-36f3ee8b128a_characterizeImage_1124831_24.stdout parsl.dataflow.dflow INFO: Standard error for task 517 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124831/3af21e47-c9d6-470d-bac3-36f3ee8b128a_characterizeImage_1124831_24.stderr parsl.dataflow.dflow INFO: Task 737 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78b3844274f3f79d338d0c6398995b61 with result from task 737 parsl.dataflow.dflow DEBUG: Task 737 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/b75673da-fc6e-4c40-a5e3-428dc2d74bc0_calibrate_1124828_38.stdout parsl.dataflow.dflow INFO: Standard error for task 737 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/b75673da-fc6e-4c40-a5e3-428dc2d74bc0_calibrate_1124828_38.stderr parsl.dataflow.dflow INFO: Task 386 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ccf1d1757c6939b683377b21d2a5a5ff with result from task 386 parsl.dataflow.dflow DEBUG: Task 386 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/5f6480c5-35fd-494b-af05-0febbcb48064_calibrate_1124828_48.stdout parsl.dataflow.dflow INFO: Standard error for task 386 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/5f6480c5-35fd-494b-af05-0febbcb48064_calibrate_1124828_48.stderr parsl.dataflow.dflow INFO: Task 317 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e190e36c70e6fb5d83b687d5456b2e0a with result from task 317 parsl.dataflow.dflow DEBUG: Task 317 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4d2fb781-ade2-4b80-b3c4-7f6e36953563_calibrate_1124831_57.stdout parsl.dataflow.dflow INFO: Standard error for task 317 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4d2fb781-ade2-4b80-b3c4-7f6e36953563_calibrate_1124831_57.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:36 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 880 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2724e7adeb015e5dced01410d7acc428 with result from task 880 parsl.dataflow.dflow DEBUG: Task 880 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 881 has memoization hash 9b631e9a3cce320a821f2aa49bc7c00c parsl.dataflow.memoization INFO: Task 881 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960ac00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 881 try 0 launched on executor multi with executor id 870 parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/70ac579c-d8d3-4430-a213-a37cf4ea14fb_calibrate_1124704_54.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/70ac579c-d8d3-4430-a213-a37cf4ea14fb_calibrate_1124704_54.stderr parsl.dataflow.dflow INFO: Standard output for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7f9ded49-ad0f-41ba-bf41-ace6f7d599f7_characterizeImage_1124704_54.stdout parsl.dataflow.dflow INFO: Standard error for task 880 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/7f9ded49-ad0f-41ba-bf41-ace6f7d599f7_characterizeImage_1124704_54.stderr parsl.dataflow.dflow INFO: Task 754 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 205400159d6ab994bee336b47a6fa6bb with result from task 754 parsl.dataflow.dflow DEBUG: Task 754 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 755 has memoization hash d80f3fa96dd42027bf8af5ddda15874b parsl.dataflow.memoization INFO: Task 755 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977e020> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 755 try 0 launched on executor multi with executor id 871 parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/86c4e160-796c-4069-8071-fd2e86cccf19_calibrate_1124704_38.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/86c4e160-796c-4069-8071-fd2e86cccf19_calibrate_1124704_38.stderr parsl.dataflow.dflow INFO: Standard output for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b92cb915-c845-4826-a96c-c50fab7d46a3_characterizeImage_1124704_38.stdout parsl.dataflow.dflow INFO: Standard error for task 754 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/b92cb915-c845-4826-a96c-c50fab7d46a3_characterizeImage_1124704_38.stderr parsl.dataflow.dflow INFO: Task 760 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry edcf7ef07e7131fc231c5082c3281f2d with result from task 760 parsl.dataflow.dflow DEBUG: Task 760 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 761 has memoization hash da2acf08c3978089827c89def71446f2 parsl.dataflow.memoization INFO: Task 761 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977e700> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 761 try 0 launched on executor multi with executor id 872 parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8800fe4f-1870-401a-a66c-6d4f67f9f11b_calibrate_1124704_51.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8800fe4f-1870-401a-a66c-6d4f67f9f11b_calibrate_1124704_51.stderr parsl.dataflow.dflow INFO: Standard output for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/ad3dd203-1a65-4a9c-ad4b-016f013d556b_characterizeImage_1124704_51.stdout parsl.dataflow.dflow INFO: Standard error for task 760 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/ad3dd203-1a65-4a9c-ad4b-016f013d556b_characterizeImage_1124704_51.stderr parsl.dataflow.dflow INFO: Task 728 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 69ce951539f7dd14c4d55575c23f89ba with result from task 728 parsl.dataflow.dflow DEBUG: Task 728 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/3e728ba3-30a7-4000-ab4f-49ed2c3e4b5f_calibrate_1124828_49.stdout parsl.dataflow.dflow INFO: Standard error for task 728 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/3e728ba3-30a7-4000-ab4f-49ed2c3e4b5f_calibrate_1124828_49.stderr parsl.dataflow.dflow INFO: Task 904 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0bca5306f5515e9f4843afc481698d1 with result from task 904 parsl.dataflow.dflow DEBUG: Task 904 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 905 has memoization hash fb75ec92a74f9c34bedd0510023b88e4 parsl.dataflow.memoization INFO: Task 905 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529667920> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 905 try 0 launched on executor multi with executor id 873 parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/521c2687-a0ad-43a3-9c19-0ad8ef9f522b_calibrate_1124707_1.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/521c2687-a0ad-43a3-9c19-0ad8ef9f522b_calibrate_1124707_1.stderr parsl.dataflow.dflow INFO: Standard output for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8211d449-7a80-4243-9356-6c3a823807e4_characterizeImage_1124707_1.stdout parsl.dataflow.dflow INFO: Standard error for task 904 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8211d449-7a80-4243-9356-6c3a823807e4_characterizeImage_1124707_1.stderr parsl.dataflow.dflow INFO: Task 764 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c86d90c1d496cc724a123ba967aa3fc0 with result from task 764 parsl.dataflow.dflow DEBUG: Task 764 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a964d103-2452-4dd7-b35c-5bcab14e478c_calibrate_1124828_40.stdout parsl.dataflow.dflow INFO: Standard error for task 764 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a964d103-2452-4dd7-b35c-5bcab14e478c_calibrate_1124828_40.stderr parsl.dataflow.dflow INFO: Task 658 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f493f5a01883ad22b7605c8cedafd302 with result from task 658 parsl.dataflow.dflow DEBUG: Task 658 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 659 has memoization hash 5144703371afbfacad41091eaafc58a0 parsl.dataflow.memoization INFO: Task 659 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297380e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 659 try 0 launched on executor multi with executor id 874 parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/4eaf136c-c348-4fd8-bebc-330e92cc1512_calibrate_1124704_30.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/4eaf136c-c348-4fd8-bebc-330e92cc1512_calibrate_1124704_30.stderr parsl.dataflow.dflow INFO: Standard output for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/c8bcd4db-c7b5-498f-ac27-a5b26d09cfb3_characterizeImage_1124704_30.stdout parsl.dataflow.dflow INFO: Standard error for task 658 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/c8bcd4db-c7b5-498f-ac27-a5b26d09cfb3_characterizeImage_1124704_30.stderr parsl.dataflow.dflow INFO: Task 508 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6dac1d57653fb914574b6e974e4abe54 with result from task 508 parsl.dataflow.dflow DEBUG: Task 508 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 509 has memoization hash 66cc1d675ec5f00c3bb6e6b902931c29 parsl.dataflow.memoization INFO: Task 509 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529851080> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 509 try 0 launched on executor multi with executor id 875 parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/5ceab459-6ca7-48fc-986f-fb9684d491e3_calibrate_1124704_3.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/5ceab459-6ca7-48fc-986f-fb9684d491e3_calibrate_1124704_3.stderr parsl.dataflow.dflow INFO: Standard output for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/9c9b4813-8367-445d-a76f-b74d85dd2992_characterizeImage_1124704_3.stdout parsl.dataflow.dflow INFO: Standard error for task 508 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/9c9b4813-8367-445d-a76f-b74d85dd2992_characterizeImage_1124704_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:37 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 190 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5128426c8eec1d936c142d1ff749641a with result from task 190 parsl.dataflow.dflow DEBUG: Task 190 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 191 has memoization hash 8b832784b3a78c952c2ce11ea220f855 parsl.dataflow.memoization INFO: Task 191 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adad6c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 191 try 0 launched on executor multi with executor id 876 parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/64017345-fc07-4b33-9e6c-e7959f03b047_calibrate_1124707_27.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/64017345-fc07-4b33-9e6c-e7959f03b047_calibrate_1124707_27.stderr parsl.dataflow.dflow INFO: Standard output for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/bd8f5ea9-1bab-4d98-8530-2466e432e79f_characterizeImage_1124707_27.stdout parsl.dataflow.dflow INFO: Standard error for task 190 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/bd8f5ea9-1bab-4d98-8530-2466e432e79f_characterizeImage_1124707_27.stderr parsl.dataflow.dflow INFO: Task 358 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0fa3744afcf3d714d3f33e4a5a347a9f with result from task 358 parsl.dataflow.dflow DEBUG: Task 358 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 359 has memoization hash 5774cb67f4f2d0dfbd687805402f0ddb parsl.dataflow.memoization INFO: Task 359 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297e8680> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 359 try 0 launched on executor multi with executor id 877 parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/290bb44a-f309-4176-98eb-018c6cd98529_calibrate_1124710_43.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/290bb44a-f309-4176-98eb-018c6cd98529_calibrate_1124710_43.stderr parsl.dataflow.dflow INFO: Standard output for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/0218f2db-e2af-4b5f-9ede-1cc1aab1d036_characterizeImage_1124710_43.stdout parsl.dataflow.dflow INFO: Standard error for task 358 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/0218f2db-e2af-4b5f-9ede-1cc1aab1d036_characterizeImage_1124710_43.stderr parsl.dataflow.dflow INFO: Task 671 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 83119de6c2148debaf9fed92ac4ae03b with result from task 671 parsl.dataflow.dflow DEBUG: Task 671 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/2f96bf99-8c50-4cc3-baed-81d254d7aa41_calibrate_1124828_47.stdout parsl.dataflow.dflow INFO: Standard error for task 671 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/2f96bf99-8c50-4cc3-baed-81d254d7aa41_calibrate_1124828_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:38 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 569 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0907d0a6fb29f20ea8885e53bf8561e6 with result from task 569 parsl.dataflow.dflow DEBUG: Task 569 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/983e0f51-f60f-4998-b8a6-a300c0a2203d_calibrate_1124831_37.stdout parsl.dataflow.dflow INFO: Standard error for task 569 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/983e0f51-f60f-4998-b8a6-a300c0a2203d_calibrate_1124831_37.stderr parsl.dataflow.dflow INFO: Task 178 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3480215388c717d66266e0f34214fdda with result from task 178 parsl.dataflow.dflow DEBUG: Task 178 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 179 has memoization hash 7f4b8651b13755a60530eb60e3d38bbc parsl.dataflow.memoization INFO: Task 179 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adaca40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 179 try 0 launched on executor multi with executor id 878 parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/26a723c0-e463-4053-a0c5-b28e5091fcbb_calibrate_1124704_60.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/26a723c0-e463-4053-a0c5-b28e5091fcbb_calibrate_1124704_60.stderr parsl.dataflow.dflow INFO: Standard output for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/a62a1edc-bc4e-4695-99f5-0f7e442f58c3_characterizeImage_1124704_60.stdout parsl.dataflow.dflow INFO: Standard error for task 178 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/a62a1edc-bc4e-4695-99f5-0f7e442f58c3_characterizeImage_1124704_60.stderr parsl.dataflow.dflow INFO: Task 259 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 61806bd8c3abe7cba5585cfcc6f892df with result from task 259 parsl.dataflow.dflow DEBUG: Task 259 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 260 has memoization hash 0c83fcc8f09aad46676ba1597a077bf7 parsl.dataflow.memoization INFO: Task 260 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9ee0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 260 try 0 launched on executor multi with executor id 879 parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f4fba208-e15e-4bc6-a34b-a41a9e71105c_calibrate_1124828_27.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f4fba208-e15e-4bc6-a34b-a41a9e71105c_calibrate_1124828_27.stderr parsl.dataflow.dflow INFO: Standard output for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/eecb1a2a-c324-4bdc-9980-6237c9de2751_characterizeImage_1124828_27.stdout parsl.dataflow.dflow INFO: Standard error for task 259 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124828/eecb1a2a-c324-4bdc-9980-6237c9de2751_characterizeImage_1124828_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:39 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 482 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7bdd787bfb382408c5f5306cac818ed with result from task 482 parsl.dataflow.dflow DEBUG: Task 482 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/b5796c22-e0cc-4ffd-aabc-08cb413631c5_calibrate_1124831_16.stdout parsl.dataflow.dflow INFO: Standard error for task 482 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/b5796c22-e0cc-4ffd-aabc-08cb413631c5_calibrate_1124831_16.stderr parsl.dataflow.dflow INFO: Task 157 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbcac6eb2eb32764a1f45a1acd6f2031 with result from task 157 parsl.dataflow.dflow DEBUG: Task 157 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 158 has memoization hash 8a4a422a9393f81fbfdd4c1b83645d17 parsl.dataflow.memoization INFO: Task 158 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8f420> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 158 try 0 launched on executor multi with executor id 880 parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/cdd0acc1-1054-437b-94a1-c6ffad16d57d_calibrate_1124710_56.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/cdd0acc1-1054-437b-94a1-c6ffad16d57d_calibrate_1124710_56.stderr parsl.dataflow.dflow INFO: Standard output for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1b53a577-184f-4c6e-8ac3-4bd1e85233bc_characterizeImage_1124710_56.stdout parsl.dataflow.dflow INFO: Standard error for task 157 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/1b53a577-184f-4c6e-8ac3-4bd1e85233bc_characterizeImage_1124710_56.stderr parsl.dataflow.dflow INFO: Task 695 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0712b93e757c9dd292afa885c2d9c98b with result from task 695 parsl.dataflow.dflow DEBUG: Task 695 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/d745d679-ff4c-4160-ab35-2b2425e7654c_calibrate_1124828_25.stdout parsl.dataflow.dflow INFO: Standard error for task 695 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/d745d679-ff4c-4160-ab35-2b2425e7654c_calibrate_1124828_25.stderr parsl.dataflow.dflow INFO: Task 134 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c4898231263cc8ecbd7504e6a9b7e95a with result from task 134 parsl.dataflow.dflow DEBUG: Task 134 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/e14e416e-ed94-4a9a-a12a-653c13e18d01_calibrate_1124831_20.stdout parsl.dataflow.dflow INFO: Standard error for task 134 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/e14e416e-ed94-4a9a-a12a-653c13e18d01_calibrate_1124831_20.stderr parsl.dataflow.dflow INFO: Task 824 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8c23964c1b1b7c0b0db1ac0a454b3425 with result from task 824 parsl.dataflow.dflow DEBUG: Task 824 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/751bb54d-07a4-4f51-89be-9c6043833420_calibrate_1124831_12.stdout parsl.dataflow.dflow INFO: Standard error for task 824 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/751bb54d-07a4-4f51-89be-9c6043833420_calibrate_1124831_12.stderr parsl.dataflow.dflow INFO: Task 700 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6beaafd98a2fdac448778b00bddfe22b with result from task 700 parsl.dataflow.dflow DEBUG: Task 700 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 701 has memoization hash 37dc7f1b17045bd9031815ebd160fe4f parsl.dataflow.memoization INFO: Task 701 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973b9c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 701 try 0 launched on executor multi with executor id 881 parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/aea468ce-e71e-4fd3-ae88-53be4400f6a3_calibrate_1124704_16.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/aea468ce-e71e-4fd3-ae88-53be4400f6a3_calibrate_1124704_16.stderr parsl.dataflow.dflow INFO: Standard output for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/641756cc-bb83-4115-abf7-adfe78b7454c_characterizeImage_1124704_16.stdout parsl.dataflow.dflow INFO: Standard error for task 700 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/641756cc-bb83-4115-abf7-adfe78b7454c_characterizeImage_1124704_16.stderr parsl.dataflow.dflow INFO: Task 448 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c88745a4ce2964340d0c6a78958aea7c with result from task 448 parsl.dataflow.dflow DEBUG: Task 448 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 449 has memoization hash 595b3337feab9bf4ebbf7d935cc996de parsl.dataflow.memoization INFO: Task 449 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981e660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 449 try 0 launched on executor multi with executor id 882 parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/9734fcd8-259c-42a0-8361-744c66ace823_calibrate_1124707_54.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/9734fcd8-259c-42a0-8361-744c66ace823_calibrate_1124707_54.stderr parsl.dataflow.dflow INFO: Standard output for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3a3e8de2-f12e-4865-b52a-5156b32f3032_characterizeImage_1124707_54.stdout parsl.dataflow.dflow INFO: Standard error for task 448 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/3a3e8de2-f12e-4865-b52a-5156b32f3032_characterizeImage_1124707_54.stderr parsl.dataflow.dflow INFO: Task 718 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 35f0dc95bb387774e63407ae33e3a36e with result from task 718 parsl.dataflow.dflow DEBUG: Task 718 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 719 has memoization hash 7c9a4c025a51a358ed10767b2e0eaed5 parsl.dataflow.memoization INFO: Task 719 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977c4a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 719 try 0 launched on executor multi with executor id 883 parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/edfff04a-74b3-49c5-9a5f-1d8f048da000_calibrate_1124707_13.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/edfff04a-74b3-49c5-9a5f-1d8f048da000_calibrate_1124707_13.stderr parsl.dataflow.dflow INFO: Standard output for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/fa5cbde5-8361-4567-8314-b66bcec04cd3_characterizeImage_1124707_13.stdout parsl.dataflow.dflow INFO: Standard error for task 718 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/fa5cbde5-8361-4567-8314-b66bcec04cd3_characterizeImage_1124707_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:40 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 869 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cea1991d5fcef5cdb6016bbe38d2c67 with result from task 869 parsl.dataflow.dflow DEBUG: Task 869 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/0da66c8f-37a7-41a5-bf52-ca7376e61120_calibrate_1124831_26.stdout parsl.dataflow.dflow INFO: Standard error for task 869 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/0da66c8f-37a7-41a5-bf52-ca7376e61120_calibrate_1124831_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:41 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 830 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7ae4ffb25128aba3b5de5f6d51ffcbd with result from task 830 parsl.dataflow.dflow DEBUG: Task 830 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8cb8878b-db49-45a0-85fe-6fc718390e40_calibrate_1124828_56.stdout parsl.dataflow.dflow INFO: Standard error for task 830 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/8cb8878b-db49-45a0-85fe-6fc718390e40_calibrate_1124828_56.stderr parsl.dataflow.dflow INFO: Task 661 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb7cf0d0b55e390bc1ea1ce7667dfeef with result from task 661 parsl.dataflow.dflow DEBUG: Task 661 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 662 has memoization hash c85cf9ac71c7b3b1530c9246a4d7b840 parsl.dataflow.memoization INFO: Task 662 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529738540> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 662 try 0 launched on executor multi with executor id 884 parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/926b33b3-5a43-4374-9ee2-791a03c29e24_calibrate_1124710_49.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/926b33b3-5a43-4374-9ee2-791a03c29e24_calibrate_1124710_49.stderr parsl.dataflow.dflow INFO: Standard output for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e00ff511-24e2-45fc-91b7-f241c62d1d13_characterizeImage_1124710_49.stdout parsl.dataflow.dflow INFO: Standard error for task 661 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/e00ff511-24e2-45fc-91b7-f241c62d1d13_characterizeImage_1124710_49.stderr parsl.dataflow.dflow INFO: Task 161 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 08e0b42d0c8cc4491b86db46429d3eb3 with result from task 161 parsl.dataflow.dflow DEBUG: Task 161 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/40a352d0-8a16-4ae1-9d03-4656b985f995_calibrate_1124828_62.stdout parsl.dataflow.dflow INFO: Standard error for task 161 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/40a352d0-8a16-4ae1-9d03-4656b985f995_calibrate_1124828_62.stderr parsl.dataflow.dflow INFO: Task 194 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 798e310dc783be490f845045fde7386a with result from task 194 parsl.dataflow.dflow DEBUG: Task 194 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/96d86ae5-f619-40fe-a493-b95fb5fe8a8e_calibrate_1124828_11.stdout parsl.dataflow.dflow INFO: Standard error for task 194 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/96d86ae5-f619-40fe-a493-b95fb5fe8a8e_calibrate_1124828_11.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:42 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 913 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e6890062b8db0195580eae54026c9866 with result from task 913 parsl.dataflow.dflow DEBUG: Task 913 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 914 has memoization hash db04dcf7e14d5f1bf067a941127d92ea parsl.dataflow.memoization INFO: Task 914 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296645e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 914 try 0 launched on executor multi with executor id 885 parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ef82e583-965b-4e8d-a7bd-716e780e836b_calibrate_1124710_57.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ef82e583-965b-4e8d-a7bd-716e780e836b_calibrate_1124710_57.stderr parsl.dataflow.dflow INFO: Standard output for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/53a1fe1d-cd0d-4002-9270-9c244ac4fba6_characterizeImage_1124710_57.stdout parsl.dataflow.dflow INFO: Standard error for task 913 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/53a1fe1d-cd0d-4002-9270-9c244ac4fba6_characterizeImage_1124710_57.stderr parsl.dataflow.dflow INFO: Task 247 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c1c2fffbc4a99036423359f5e73ace10 with result from task 247 parsl.dataflow.dflow DEBUG: Task 247 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 248 has memoization hash ae13f9c21ed608b1311659755659794d parsl.dataflow.memoization INFO: Task 248 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9260> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 248 try 0 launched on executor multi with executor id 886 parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/2e1f458a-47af-48f9-9abb-3b1f89fc0b7f_calibrate_1124704_40.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/2e1f458a-47af-48f9-9abb-3b1f89fc0b7f_calibrate_1124704_40.stderr parsl.dataflow.dflow INFO: Standard output for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/0171fa6b-fa0a-460f-9ea0-66edf31be47b_characterizeImage_1124704_40.stdout parsl.dataflow.dflow INFO: Standard error for task 247 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/0171fa6b-fa0a-460f-9ea0-66edf31be47b_characterizeImage_1124704_40.stderr parsl.dataflow.dflow INFO: Task 814 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab981d1f8a79c3fd8430c24f4f2939a7 with result from task 814 parsl.dataflow.dflow DEBUG: Task 814 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 815 has memoization hash 558c898a14b0e1214b132fef4cb80954 parsl.dataflow.memoization INFO: Task 815 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c9c60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 815 try 0 launched on executor multi with executor id 887 parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1be2fec9-b591-41a2-900d-926b370f2507_calibrate_1124710_38.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1be2fec9-b591-41a2-900d-926b370f2507_calibrate_1124710_38.stderr parsl.dataflow.dflow INFO: Standard output for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b5f5bbb4-aef5-4223-a9f3-2a6fd12045b8_characterizeImage_1124710_38.stdout parsl.dataflow.dflow INFO: Standard error for task 814 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b5f5bbb4-aef5-4223-a9f3-2a6fd12045b8_characterizeImage_1124710_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:43 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 709 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd2f30ada4da8fa7180068b3e534711f with result from task 709 parsl.dataflow.dflow DEBUG: Task 709 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 710 has memoization hash 673928048421be2f4a46d33acf44e959 parsl.dataflow.memoization INFO: Task 710 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973b060> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 710 try 0 launched on executor multi with executor id 888 parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/84967446-06d1-4edd-9d06-c1714e629f0b_calibrate_1124707_33.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/84967446-06d1-4edd-9d06-c1714e629f0b_calibrate_1124707_33.stderr parsl.dataflow.dflow INFO: Standard output for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/5c6b30f2-e105-4222-abb7-3c9554b7fd8a_characterizeImage_1124707_33.stdout parsl.dataflow.dflow INFO: Standard error for task 709 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/5c6b30f2-e105-4222-abb7-3c9554b7fd8a_characterizeImage_1124707_33.stderr parsl.dataflow.dflow INFO: Task 253 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fb11575da7b75e7c31cac52f3dc26b1 with result from task 253 parsl.dataflow.dflow DEBUG: Task 253 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 254 has memoization hash d9f8a54176a625e80d058273c5b5eab1 parsl.dataflow.memoization INFO: Task 254 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a98a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 254 try 0 launched on executor multi with executor id 889 parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/75fe4f29-b18b-4b69-ace4-f51a7b7b9a92_calibrate_1124704_18.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/75fe4f29-b18b-4b69-ace4-f51a7b7b9a92_calibrate_1124704_18.stderr parsl.dataflow.dflow INFO: Standard output for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/ab44db52-14ac-4081-afb6-8174c7a2a15b_characterizeImage_1124704_18.stdout parsl.dataflow.dflow INFO: Standard error for task 253 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/ab44db52-14ac-4081-afb6-8174c7a2a15b_characterizeImage_1124704_18.stderr parsl.dataflow.dflow INFO: Task 176 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 67043850a5958b0c50b31aaf1440eb6b with result from task 176 parsl.dataflow.dflow DEBUG: Task 176 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/349f702f-1574-4dab-8f4e-55f4b921e9ed_calibrate_1124831_31.stdout parsl.dataflow.dflow INFO: Standard error for task 176 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/349f702f-1574-4dab-8f4e-55f4b921e9ed_calibrate_1124831_31.stderr parsl.dataflow.dflow INFO: Task 487 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7a495e79409ed28547de4ae69537376 with result from task 487 parsl.dataflow.dflow DEBUG: Task 487 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 488 has memoization hash 78f889c3f0d8f61fcdfa790bb7f0021e parsl.dataflow.memoization INFO: Task 488 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529850900> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 488 try 0 launched on executor multi with executor id 890 parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/6579d12f-21e2-4f18-b564-f123bbf035cc_calibrate_1124707_51.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/6579d12f-21e2-4f18-b564-f123bbf035cc_calibrate_1124707_51.stderr parsl.dataflow.dflow INFO: Standard output for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/08ad96ab-3f9b-4eb0-b2df-ed72374bd680_characterizeImage_1124707_51.stdout parsl.dataflow.dflow INFO: Standard error for task 487 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/08ad96ab-3f9b-4eb0-b2df-ed72374bd680_characterizeImage_1124707_51.stderr parsl.dataflow.dflow INFO: Task 205 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 16e5b18a9b57885d8f9deeaf0bb68264 with result from task 205 parsl.dataflow.dflow DEBUG: Task 205 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 206 has memoization hash 22e0072dbee3eef026c12c865902521e parsl.dataflow.memoization INFO: Task 206 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adae660> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 206 try 0 launched on executor multi with executor id 891 parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/48caa663-0ac4-4e9b-9edd-83ebff6c7133_calibrate_1124704_28.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/48caa663-0ac4-4e9b-9edd-83ebff6c7133_calibrate_1124704_28.stderr parsl.dataflow.dflow INFO: Standard output for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/3ef2af89-1b9c-4224-9188-73b3633c1c2f_characterizeImage_1124704_28.stdout parsl.dataflow.dflow INFO: Standard error for task 205 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/3ef2af89-1b9c-4224-9188-73b3633c1c2f_characterizeImage_1124704_28.stderr parsl.dataflow.dflow INFO: Task 472 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0e02f6694082053c0d2c48eb9dc3ab2e with result from task 472 parsl.dataflow.dflow DEBUG: Task 472 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 473 has memoization hash 370e63755df3047ab63629ebc81109da parsl.dataflow.memoization INFO: Task 473 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981fce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 473 try 0 launched on executor multi with executor id 892 parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/5f12ade6-fdff-4c93-808b-d7c41cc9a482_calibrate_1124710_47.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/5f12ade6-fdff-4c93-808b-d7c41cc9a482_calibrate_1124710_47.stderr parsl.dataflow.dflow INFO: Standard output for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/70359c08-b1e9-4bea-b5e9-0b93ed4bd82f_characterizeImage_1124710_47.stdout parsl.dataflow.dflow INFO: Standard error for task 472 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/70359c08-b1e9-4bea-b5e9-0b93ed4bd82f_characterizeImage_1124710_47.stderr parsl.dataflow.dflow INFO: Task 467 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0076b71bd215121b76d562b53336b70 with result from task 467 parsl.dataflow.dflow DEBUG: Task 467 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/fe434a0b-af41-4be4-9d2f-9ec19d778168_calibrate_1124831_56.stdout parsl.dataflow.dflow INFO: Standard error for task 467 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/fe434a0b-af41-4be4-9d2f-9ec19d778168_calibrate_1124831_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:46 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 506 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b61c2943b18fb9f21bbc6b5a4c7273b with result from task 506 parsl.dataflow.dflow DEBUG: Task 506 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/73c83ca9-42d0-40b3-afb0-e5dfcf2ca78d_calibrate_1124831_4.stdout parsl.dataflow.dflow INFO: Standard error for task 506 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/73c83ca9-42d0-40b3-afb0-e5dfcf2ca78d_calibrate_1124831_4.stderr parsl.dataflow.dflow INFO: Task 149 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f0cadfc89c98672f1be71deb256abdec with result from task 149 parsl.dataflow.dflow DEBUG: Task 149 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4f4d3843-8adf-4155-9d57-84bdffc92437_calibrate_1124831_21.stdout parsl.dataflow.dflow INFO: Standard error for task 149 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4f4d3843-8adf-4155-9d57-84bdffc92437_calibrate_1124831_21.stderr parsl.dataflow.dflow INFO: Task 208 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49c2953b66edf49ff1b234e1d7056940 with result from task 208 parsl.dataflow.dflow DEBUG: Task 208 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 209 has memoization hash 179ff4acff7366e74408592d697aac43 parsl.dataflow.memoization INFO: Task 209 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adae980> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 209 try 0 launched on executor multi with executor id 893 parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/cf783236-09f4-4afb-af80-b673ab1a6b41_calibrate_1124710_46.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/cf783236-09f4-4afb-af80-b673ab1a6b41_calibrate_1124710_46.stderr parsl.dataflow.dflow INFO: Standard output for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/16595c8b-e7d3-46cf-b52d-1f3e5b90345e_characterizeImage_1124710_46.stdout parsl.dataflow.dflow INFO: Standard error for task 208 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/16595c8b-e7d3-46cf-b52d-1f3e5b90345e_characterizeImage_1124710_46.stderr parsl.dataflow.dflow INFO: Task 871 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b58f18a261b2b86debd8a7d28f6ece21 with result from task 871 parsl.dataflow.dflow DEBUG: Task 871 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 872 has memoization hash 684a572f5d0ab893a5971e3d04f3d8b5 parsl.dataflow.memoization INFO: Task 872 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960a2a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 872 try 0 launched on executor multi with executor id 894 parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/f1380197-c011-4826-978b-1d8b3b4b8b8a_calibrate_1124704_15.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/f1380197-c011-4826-978b-1d8b3b4b8b8a_calibrate_1124704_15.stderr parsl.dataflow.dflow INFO: Standard output for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/0d7ee9c1-cf96-4649-8fb0-0f16ed7964cf_characterizeImage_1124704_15.stdout parsl.dataflow.dflow INFO: Standard error for task 871 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/0d7ee9c1-cf96-4649-8fb0-0f16ed7964cf_characterizeImage_1124704_15.stderr parsl.dataflow.dflow INFO: Task 901 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8bcc9ee5066093ebe9ad1a136b9ddf1e with result from task 901 parsl.dataflow.dflow DEBUG: Task 901 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 902 has memoization hash 5dc8d02be02b5127cd85bae5e3bb279a parsl.dataflow.memoization INFO: Task 902 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296640e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 902 try 0 launched on executor multi with executor id 895 parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/fdec7c70-6bf5-4f35-9ac0-567c652f5bcb_calibrate_1124704_23.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/fdec7c70-6bf5-4f35-9ac0-567c652f5bcb_calibrate_1124704_23.stderr parsl.dataflow.dflow INFO: Standard output for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/46eb9269-cc5f-431c-9006-8d923e856060_characterizeImage_1124704_23.stdout parsl.dataflow.dflow INFO: Standard error for task 901 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/46eb9269-cc5f-431c-9006-8d923e856060_characterizeImage_1124704_23.stderr parsl.dataflow.dflow INFO: Task 197 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 359b6e194f6bd54372a50902fc8770d5 with result from task 197 parsl.dataflow.dflow DEBUG: Task 197 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/f9f424a7-bff6-4c43-b6ae-4682f72c2540_calibrate_1124831_34.stdout parsl.dataflow.dflow INFO: Standard error for task 197 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/f9f424a7-bff6-4c43-b6ae-4682f72c2540_calibrate_1124831_34.stderr parsl.dataflow.dflow INFO: Task 779 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a4c92645a54c683fdccfd2e31191600 with result from task 779 parsl.dataflow.dflow DEBUG: Task 779 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/651bbdb2-a798-4596-8296-7942fa3c1c40_calibrate_1124828_3.stdout parsl.dataflow.dflow INFO: Standard error for task 779 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/651bbdb2-a798-4596-8296-7942fa3c1c40_calibrate_1124828_3.stderr parsl.dataflow.dflow INFO: Task 793 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7403a737f5906e6b7466de36ffac60ac with result from task 793 parsl.dataflow.dflow DEBUG: Task 793 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 794 has memoization hash c7ee6db9b3b197f677de0f8f94295c29 parsl.dataflow.memoization INFO: Task 794 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c8e00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 794 try 0 launched on executor multi with executor id 896 parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/3130fa52-7612-45de-98ec-2b296ee56466_calibrate_1124707_45.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/3130fa52-7612-45de-98ec-2b296ee56466_calibrate_1124707_45.stderr parsl.dataflow.dflow INFO: Standard output for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/cb0357eb-001e-45a8-933b-d478e4a171be_characterizeImage_1124707_45.stdout parsl.dataflow.dflow INFO: Standard error for task 793 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/cb0357eb-001e-45a8-933b-d478e4a171be_characterizeImage_1124707_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:48 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 769 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fd6a56d3cb50e6534e9c8a515ec0881 with result from task 769 parsl.dataflow.dflow DEBUG: Task 769 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 770 has memoization hash e87cf48b3eb63279c6c3f87b84f0ad71 parsl.dataflow.memoization INFO: Task 770 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977f6a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 770 try 0 launched on executor multi with executor id 897 parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/2efc6732-25ab-49f8-9759-ec06e1886f24_calibrate_1124704_59.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/2efc6732-25ab-49f8-9759-ec06e1886f24_calibrate_1124704_59.stderr parsl.dataflow.dflow INFO: Standard output for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/5627f992-3f4f-419b-97cd-88d9cd8ef0f6_characterizeImage_1124704_59.stdout parsl.dataflow.dflow INFO: Standard error for task 769 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/5627f992-3f4f-419b-97cd-88d9cd8ef0f6_characterizeImage_1124704_59.stderr parsl.dataflow.dflow INFO: Task 854 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 47fa0a130cd4c470960bf72dfea7c6c0 with result from task 854 parsl.dataflow.dflow DEBUG: Task 854 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/7bb2b433-1a01-4454-858f-9c787e55fdc2_calibrate_1124828_44.stdout parsl.dataflow.dflow INFO: Standard error for task 854 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/7bb2b433-1a01-4454-858f-9c787e55fdc2_calibrate_1124828_44.stderr parsl.dataflow.dflow INFO: Task 875 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 265e1149101cf1be376f0b23dd22c23a with result from task 875 parsl.dataflow.dflow DEBUG: Task 875 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/2900821b-bb9a-44a0-ac2f-cac8c4927755_calibrate_1124831_38.stdout parsl.dataflow.dflow INFO: Standard error for task 875 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/2900821b-bb9a-44a0-ac2f-cac8c4927755_calibrate_1124831_38.stderr parsl.dataflow.dflow INFO: Task 707 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3bcbd7904d8d46109ae27da4fe848729 with result from task 707 parsl.dataflow.dflow DEBUG: Task 707 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/10d5a4cd-3ef6-43ce-bf62-04d517af491d_calibrate_1124831_15.stdout parsl.dataflow.dflow INFO: Standard error for task 707 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/10d5a4cd-3ef6-43ce-bf62-04d517af491d_calibrate_1124831_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:49 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 491 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 14095883208391a74e952a8c8d34cda3 with result from task 491 parsl.dataflow.dflow DEBUG: Task 491 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3bace157-5893-430f-8979-91b5084460f8_calibrate_1124831_32.stdout parsl.dataflow.dflow INFO: Standard error for task 491 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/3bace157-5893-430f-8979-91b5084460f8_calibrate_1124831_32.stderr parsl.dataflow.dflow INFO: Task 860 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b44466eb37f8671b06bdd3388fe91459 with result from task 860 parsl.dataflow.dflow DEBUG: Task 860 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/c1df79fd-1d14-4699-941f-0c366b7d263a_calibrate_1124828_52.stdout parsl.dataflow.dflow INFO: Standard error for task 860 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/c1df79fd-1d14-4699-941f-0c366b7d263a_calibrate_1124828_52.stderr parsl.dataflow.dflow INFO: Task 80 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8598b0bce161afae7bea16b3d86f94f6 with result from task 80 parsl.dataflow.dflow DEBUG: Task 80 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/38fb9a8a-c24d-419f-8117-c238ae95090b_calibrate_1124710_18.stdout parsl.dataflow.dflow INFO: Standard error for task 80 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/38fb9a8a-c24d-419f-8117-c238ae95090b_calibrate_1124710_18.stderr parsl.dataflow.dflow INFO: Task 884 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8c1943fbc76798a70446079045ff5f1 with result from task 884 parsl.dataflow.dflow DEBUG: Task 884 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/ad35a760-797d-42b7-909a-fda183dc5e26_calibrate_1124828_22.stdout parsl.dataflow.dflow INFO: Standard error for task 884 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/ad35a760-797d-42b7-909a-fda183dc5e26_calibrate_1124828_22.stderr parsl.dataflow.dflow INFO: Task 512 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42a357a8a551d81feb79ac877d2998f3 with result from task 512 parsl.dataflow.dflow DEBUG: Task 512 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/004749a0-39cb-4d82-84ea-cab7ea638d88_calibrate_1124831_50.stdout parsl.dataflow.dflow INFO: Standard error for task 512 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/004749a0-39cb-4d82-84ea-cab7ea638d88_calibrate_1124831_50.stderr parsl.dataflow.dflow INFO: Task 221 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 49423f01bd68bc26df48ae9e14b07391 with result from task 221 parsl.dataflow.dflow DEBUG: Task 221 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/bf836b3c-aecb-4116-b23e-f7b6d1e0a3c2_calibrate_1124828_17.stdout parsl.dataflow.dflow INFO: Standard error for task 221 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/bf836b3c-aecb-4116-b23e-f7b6d1e0a3c2_calibrate_1124828_17.stderr parsl.dataflow.dflow INFO: Task 464 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5be11f1b0a164ef3c9580f4d96822cdc with result from task 464 parsl.dataflow.dflow DEBUG: Task 464 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/0676aaba-6bc0-4a75-9faf-bed440b7d55c_calibrate_1124831_46.stdout parsl.dataflow.dflow INFO: Standard error for task 464 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/0676aaba-6bc0-4a75-9faf-bed440b7d55c_calibrate_1124831_46.stderr parsl.dataflow.dflow INFO: Task 742 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0b5eabe388ec5c19f3af78c3cff42b47 with result from task 742 parsl.dataflow.dflow DEBUG: Task 742 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 743 has memoization hash 8fe38dedcf049ddad4a3a2cc7f9a8bc6 parsl.dataflow.memoization INFO: Task 743 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977d3a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 743 try 0 launched on executor multi with executor id 898 parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/3d3bf46f-df3f-4ecb-a663-cbb89782424a_calibrate_1124704_24.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/3d3bf46f-df3f-4ecb-a663-cbb89782424a_calibrate_1124704_24.stderr parsl.dataflow.dflow INFO: Standard output for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/5bcfedb6-044c-42fe-aa88-9386712df486_characterizeImage_1124704_24.stdout parsl.dataflow.dflow INFO: Standard error for task 742 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/5bcfedb6-044c-42fe-aa88-9386712df486_characterizeImage_1124704_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:49 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 673 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7a06dd98207af8afddf6197b88b7eb8f with result from task 673 parsl.dataflow.dflow DEBUG: Task 673 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 674 has memoization hash a85c9c54af0f3ae1b9150f5ff06d9062 parsl.dataflow.memoization INFO: Task 674 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529738fe0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 674 try 0 launched on executor multi with executor id 899 parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/63c7a0f9-d8dc-4a7d-887d-ac2fd37fd1e1_calibrate_1124704_55.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/63c7a0f9-d8dc-4a7d-887d-ac2fd37fd1e1_calibrate_1124704_55.stderr parsl.dataflow.dflow INFO: Standard output for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/958ee1be-e054-4b11-9bb2-a7f9eac43bf3_characterizeImage_1124704_55.stdout parsl.dataflow.dflow INFO: Standard error for task 673 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/958ee1be-e054-4b11-9bb2-a7f9eac43bf3_characterizeImage_1124704_55.stderr parsl.dataflow.dflow INFO: Task 878 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b1f72c0df23bf72102bc8c1df2a9e2a with result from task 878 parsl.dataflow.dflow DEBUG: Task 878 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/5d2869b7-0aba-4f58-b9e8-361e5c7fd2d3_calibrate_1124828_12.stdout parsl.dataflow.dflow INFO: Standard error for task 878 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/5d2869b7-0aba-4f58-b9e8-361e5c7fd2d3_calibrate_1124828_12.stderr parsl.dataflow.dflow INFO: Task 454 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f66072f11fcb6efe967d7ab286886359 with result from task 454 parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:50 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow DEBUG: Task 454 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 455 has memoization hash 91d0135722e35aba63b9445c674019bc parsl.dataflow.memoization INFO: Task 455 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981eca0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 455 try 0 launched on executor multi with executor id 900 parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/229dcf9e-d025-4928-a897-fac59ff51cb7_calibrate_1124710_12.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/229dcf9e-d025-4928-a897-fac59ff51cb7_calibrate_1124710_12.stderr parsl.dataflow.dflow INFO: Standard output for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/18550bea-4988-4e18-8264-5e84181b03e5_characterizeImage_1124710_12.stdout parsl.dataflow.dflow INFO: Standard error for task 454 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/18550bea-4988-4e18-8264-5e84181b03e5_characterizeImage_1124710_12.stderr parsl.dataflow.dflow INFO: Task 892 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 42dbd900ff8d2c9aa6333e6338bdc0db with result from task 892 parsl.dataflow.dflow DEBUG: Task 892 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 893 has memoization hash b16e8fa2fd397240e0cdc26fe9bb4e06 parsl.dataflow.memoization INFO: Task 893 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960b880> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 893 try 0 launched on executor multi with executor id 901 parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/27c4ec59-233e-4596-93b5-bfc72dce093a_calibrate_1124707_23.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/27c4ec59-233e-4596-93b5-bfc72dce093a_calibrate_1124707_23.stderr parsl.dataflow.dflow INFO: Standard output for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/da5c7edd-e3a2-4a1f-bf9e-246c03002aa0_characterizeImage_1124707_23.stdout parsl.dataflow.dflow INFO: Standard error for task 892 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/da5c7edd-e3a2-4a1f-bf9e-246c03002aa0_characterizeImage_1124707_23.stderr parsl.dataflow.dflow INFO: Task 478 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ab1647b174fe3c0378664f114e841106 with result from task 478 parsl.dataflow.dflow DEBUG: Task 478 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 479 has memoization hash cd9df7a66c64921eb5fc0618f5113894 parsl.dataflow.memoization INFO: Task 479 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529850180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 479 try 0 launched on executor multi with executor id 902 parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/9fcb1502-5980-48df-a306-410a38b6e972_calibrate_1124704_11.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/9fcb1502-5980-48df-a306-410a38b6e972_calibrate_1124704_11.stderr parsl.dataflow.dflow INFO: Standard output for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d1cd61fb-2ff9-4ed6-9157-3e2306a7a139_characterizeImage_1124704_11.stdout parsl.dataflow.dflow INFO: Standard error for task 478 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/d1cd61fb-2ff9-4ed6-9157-3e2306a7a139_characterizeImage_1124704_11.stderr parsl.dataflow.dflow INFO: Task 104 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d8f67af7fc96a66e4152d7da7fefc1c8 with result from task 104 parsl.dataflow.dflow DEBUG: Task 104 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/c6a16d25-4e7c-44ff-8b6b-8a849503a3e7_calibrate_1124710_22.stdout parsl.dataflow.dflow INFO: Standard error for task 104 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/c6a16d25-4e7c-44ff-8b6b-8a849503a3e7_calibrate_1124710_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:51 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 721 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 89e08732052415b6815cd57395ddc430 with result from task 721 parsl.dataflow.dflow DEBUG: Task 721 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 722 has memoization hash 181e5c29cec5617400d34bf5694be04f parsl.dataflow.memoization INFO: Task 722 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 722 try 0 launched on executor multi with executor id 903 parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/65923946-7a79-4bf5-b21a-f7f6508cbd8a_calibrate_1124707_44.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/65923946-7a79-4bf5-b21a-f7f6508cbd8a_calibrate_1124707_44.stderr parsl.dataflow.dflow INFO: Standard output for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/26c51cd6-e4e5-46e9-abb2-edcf745b6574_characterizeImage_1124707_44.stdout parsl.dataflow.dflow INFO: Standard error for task 721 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/26c51cd6-e4e5-46e9-abb2-edcf745b6574_characterizeImage_1124707_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:52 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 152 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b0f2fee52eacd2f4d0dd606d68f60404 with result from task 152 parsl.dataflow.dflow DEBUG: Task 152 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a9c1bfc0-7cfd-4aa5-8d67-690a85ad8f6a_calibrate_1124828_26.stdout parsl.dataflow.dflow INFO: Standard error for task 152 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/a9c1bfc0-7cfd-4aa5-8d67-690a85ad8f6a_calibrate_1124828_26.stderr parsl.dataflow.dflow INFO: Task 502 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f27bee7ef348d7e869f146dea0a0b79a with result from task 502 parsl.dataflow.dflow DEBUG: Task 502 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 503 has memoization hash be2a83c8d42a87e1f6cc10c6b3510e99 parsl.dataflow.memoization INFO: Task 503 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529851580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 503 try 0 launched on executor multi with executor id 904 parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/0769f45c-e3b0-422a-aee7-561788fd79fb_calibrate_1124707_53.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/0769f45c-e3b0-422a-aee7-561788fd79fb_calibrate_1124707_53.stderr parsl.dataflow.dflow INFO: Standard output for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8574765d-1771-4f9c-a07d-3b066223e9bc_characterizeImage_1124707_53.stdout parsl.dataflow.dflow INFO: Standard error for task 502 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8574765d-1771-4f9c-a07d-3b066223e9bc_characterizeImage_1124707_53.stderr parsl.dataflow.dflow INFO: Task 287 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 24fc6b09138a9d93c010c39e57b35839 with result from task 287 parsl.dataflow.dflow DEBUG: Task 287 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/994626c2-63ce-4770-b229-c08990b995a1_calibrate_1124828_15.stdout parsl.dataflow.dflow INFO: Standard error for task 287 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/994626c2-63ce-4770-b229-c08990b995a1_calibrate_1124828_15.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:53 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 320 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44a195675c74d52022cbb30aed18ea81 with result from task 320 parsl.dataflow.dflow DEBUG: Task 320 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/fab2a62b-e52b-41de-a3fa-434345661961_calibrate_1124828_18.stdout parsl.dataflow.dflow INFO: Standard error for task 320 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/fab2a62b-e52b-41de-a3fa-434345661961_calibrate_1124828_18.stderr parsl.dataflow.dflow INFO: Task 200 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dfbad6d49ef53e715918ac468fd91f17 with result from task 200 parsl.dataflow.dflow DEBUG: Task 200 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/21cf64f8-5229-494f-8e69-40c05b93b165_calibrate_1124831_53.stdout parsl.dataflow.dflow INFO: Standard error for task 200 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/21cf64f8-5229-494f-8e69-40c05b93b165_calibrate_1124831_53.stderr parsl.dataflow.dflow INFO: Task 458 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 09a2f6797875026d8256cd05b08a651f with result from task 458 parsl.dataflow.dflow DEBUG: Task 458 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/6993bf89-2b0b-457a-9284-97091d643c42_calibrate_1124828_35.stdout parsl.dataflow.dflow INFO: Standard error for task 458 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/6993bf89-2b0b-457a-9284-97091d643c42_calibrate_1124828_35.stderr parsl.dataflow.dflow INFO: Task 806 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05974a8eee1b5254e9a8e0b98e51e2b9 with result from task 806 parsl.dataflow.dflow DEBUG: Task 806 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/c617d94a-0ac6-4b1f-ad1c-ce9f27e7ed95_calibrate_1124828_4.stdout parsl.dataflow.dflow INFO: Standard error for task 806 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/c617d94a-0ac6-4b1f-ad1c-ce9f27e7ed95_calibrate_1124828_4.stderr parsl.dataflow.dflow INFO: Task 470 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c3913ed39677e8a6f0a3e93557e51d2 with result from task 470 parsl.dataflow.dflow DEBUG: Task 470 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4fa65211-1768-4381-bddf-58e0bde1f57c_calibrate_1124831_54.stdout parsl.dataflow.dflow INFO: Standard error for task 470 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/4fa65211-1768-4381-bddf-58e0bde1f57c_calibrate_1124831_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:54 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 926 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ac3f67e7a9ebfcdfa0b04431dbd2a052 with result from task 926 parsl.dataflow.dflow DEBUG: Task 926 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/6e868246-72d0-457e-86fd-4541d31bde82_calibrate_1124828_7.stdout parsl.dataflow.dflow INFO: Standard error for task 926 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/6e868246-72d0-457e-86fd-4541d31bde82_calibrate_1124828_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:55 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 704 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7876dafe26d3608f8ce73f2826fc2d38 with result from task 704 parsl.dataflow.dflow DEBUG: Task 704 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/fe036f83-95fc-4c75-a70d-cc8841d35c66_calibrate_1124828_19.stdout parsl.dataflow.dflow INFO: Standard error for task 704 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/fe036f83-95fc-4c75-a70d-cc8841d35c66_calibrate_1124828_19.stderr parsl.dataflow.dflow INFO: Task 283 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 846227783859fa60a78d6a4b00ab5ac9 with result from task 283 parsl.dataflow.dflow DEBUG: Task 283 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 284 has memoization hash 3c79d2b25ae0efe7ebfac3514c3f77aa parsl.dataflow.memoization INFO: Task 284 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ab7e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 284 try 0 launched on executor multi with executor id 905 parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/20a869a7-d8fb-4770-ab0e-f4720fe60e68_calibrate_1124707_22.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/20a869a7-d8fb-4770-ab0e-f4720fe60e68_calibrate_1124707_22.stderr parsl.dataflow.dflow INFO: Standard output for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/931e8265-ac0a-4e41-82a0-4e5c9a6a6e1f_characterizeImage_1124707_22.stdout parsl.dataflow.dflow INFO: Standard error for task 283 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/931e8265-ac0a-4e41-82a0-4e5c9a6a6e1f_characterizeImage_1124707_22.stderr parsl.dataflow.dflow INFO: Task 250 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba931c6711c32550048616d6da71803e with result from task 250 parsl.dataflow.dflow DEBUG: Task 250 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 251 has memoization hash 92d548251728ada8a09f10363f50dd40 parsl.dataflow.memoization INFO: Task 251 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297a9580> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 251 try 0 launched on executor multi with executor id 906 parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/e779a88d-9134-4799-99d0-12731bd5e0c2_calibrate_1124710_44.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/e779a88d-9134-4799-99d0-12731bd5e0c2_calibrate_1124710_44.stderr parsl.dataflow.dflow INFO: Standard output for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b2df4c89-fa55-4b55-8892-e56c0f133bd8_characterizeImage_1124710_44.stdout parsl.dataflow.dflow INFO: Standard error for task 250 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/b2df4c89-fa55-4b55-8892-e56c0f133bd8_characterizeImage_1124710_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:56 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 140 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 48e9018084362d0302b86781de6d4262 with result from task 140 parsl.dataflow.dflow DEBUG: Task 140 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/0022f63b-6b87-402b-a837-4d40977f77fc_calibrate_1124710_33.stdout parsl.dataflow.dflow INFO: Standard error for task 140 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/0022f63b-6b87-402b-a837-4d40977f77fc_calibrate_1124710_33.stderr parsl.dataflow.dflow INFO: Task 521 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 05ed2e741162230bb6d1a10d42056065 with result from task 521 parsl.dataflow.dflow DEBUG: Task 521 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/d93da2a2-494f-42ed-b7fa-e142312b1f2c_calibrate_1124831_49.stdout parsl.dataflow.dflow INFO: Standard error for task 521 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/d93da2a2-494f-42ed-b7fa-e142312b1f2c_calibrate_1124831_49.stderr parsl.dataflow.dflow INFO: Task 847 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9c1a5720583214712e584a5241810446 with result from task 847 parsl.dataflow.dflow DEBUG: Task 847 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 848 has memoization hash dbb81b14303e9b5ee6ae25b4fae0d699 parsl.dataflow.memoization INFO: Task 848 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296089a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 848 try 0 launched on executor multi with executor id 907 parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/feb963bf-87e5-4cfc-a9a1-1c935a042ff9_calibrate_1124707_59.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/feb963bf-87e5-4cfc-a9a1-1c935a042ff9_calibrate_1124707_59.stderr parsl.dataflow.dflow INFO: Standard output for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/c1ecdacf-edcb-4edb-825a-c0455d829e44_characterizeImage_1124707_59.stdout parsl.dataflow.dflow INFO: Standard error for task 847 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/c1ecdacf-edcb-4edb-825a-c0455d829e44_characterizeImage_1124707_59.stderr parsl.dataflow.dflow INFO: Task 119 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d430aabf67fae363b834c9542b0bdbcd with result from task 119 parsl.dataflow.dflow DEBUG: Task 119 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7dd962a7-d977-470e-b57f-16a6913e45ec_calibrate_1124710_9.stdout parsl.dataflow.dflow INFO: Standard error for task 119 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7dd962a7-d977-470e-b57f-16a6913e45ec_calibrate_1124710_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:57 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 107 try 1 failed parsl.dataflow.dflow ERROR: Task 107 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry bcd408b2aee6cf23dfa118d4f35ab8cd with result from task 107 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/526ba917-d599-4483-8704-f932cf8e8631_calibrate_1124710_31.stdout parsl.dataflow.dflow INFO: Standard error for task 107 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/526ba917-d599-4483-8704-f932cf8e8631_calibrate_1124710_31.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 539 try 1 failed parsl.dataflow.dflow ERROR: Task 539 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry e171923504aa2c4ccbb268b2f6830fc6 with result from task 539 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/83c52c40-c27e-433c-98bf-f968d4880528_calibrate_1124707_31.stdout parsl.dataflow.dflow INFO: Standard error for task 539 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/83c52c40-c27e-433c-98bf-f968d4880528_calibrate_1124707_31.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:58 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 515 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 785f9a788ca8ecd6b0e9852958c0c8e6 with result from task 515 parsl.dataflow.dflow DEBUG: Task 515 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/a8fbb37a-55a6-46c6-9f25-07166dd5ba4b_calibrate_1124710_62.stdout parsl.dataflow.dflow INFO: Standard error for task 515 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/a8fbb37a-55a6-46c6-9f25-07166dd5ba4b_calibrate_1124710_62.stderr parsl.dataflow.dflow INFO: Task 377 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cda33b9ebd6965336cdedb4e03e9321e with result from task 377 parsl.dataflow.dflow DEBUG: Task 377 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/00d1d206-0092-461c-9548-5bd629b5b056_calibrate_1124704_26.stdout parsl.dataflow.dflow INFO: Standard error for task 377 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/00d1d206-0092-461c-9548-5bd629b5b056_calibrate_1124704_26.stderr parsl.dataflow.dflow INFO: Task 362 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a4ed41033651c86dcc9ae6e8a5c66ae3 with result from task 362 parsl.dataflow.dflow DEBUG: Task 362 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d2cbb77c-2982-4bac-bdd8-ce663118ca1e_calibrate_1124707_8.stdout parsl.dataflow.dflow INFO: Standard error for task 362 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d2cbb77c-2982-4bac-bdd8-ce663118ca1e_calibrate_1124707_8.stderr parsl.dataflow.dflow INFO: Task 116 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a1932d6d80f1a5deb0d4c2ab6cf432a1 with result from task 116 parsl.dataflow.dflow DEBUG: Task 116 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/ca9316f3-2267-40e4-ba24-2b8fea2dc9d0_calibrate_1124828_58.stdout parsl.dataflow.dflow INFO: Standard error for task 116 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/ca9316f3-2267-40e4-ba24-2b8fea2dc9d0_calibrate_1124828_58.stderr parsl.dataflow.dflow INFO: Task 845 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 80084c22b7b5e55f73eee99a269b4769 with result from task 845 parsl.dataflow.dflow DEBUG: Task 845 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1807ab7d-7594-4aee-9699-7395b6917229_calibrate_1124707_62.stdout parsl.dataflow.dflow INFO: Standard error for task 845 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1807ab7d-7594-4aee-9699-7395b6917229_calibrate_1124707_62.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:29:59 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 146 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5b1c4c29b2d431b80dabce693aef8d30 with result from task 146 parsl.dataflow.dflow DEBUG: Task 146 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/8e1623ab-99ca-4f7f-8869-48c3efe8c16d_calibrate_1124710_23.stdout parsl.dataflow.dflow INFO: Standard error for task 146 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/8e1623ab-99ca-4f7f-8869-48c3efe8c16d_calibrate_1124710_23.stderr parsl.dataflow.dflow INFO: Task 172 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0ebc61a4214ff078c5124150d650fb9 with result from task 172 parsl.dataflow.dflow DEBUG: Task 172 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 173 has memoization hash e421a711bcb804f8b9da49f6e74512de parsl.dataflow.memoization INFO: Task 173 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adac400> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 173 try 0 launched on executor multi with executor id 908 parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/609c82be-6502-4b6b-a303-f9425d098b68_calibrate_1124707_30.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/609c82be-6502-4b6b-a303-f9425d098b68_calibrate_1124707_30.stderr parsl.dataflow.dflow INFO: Standard output for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/5e46d950-66ac-4a6d-abe4-137e0d65fa9e_characterizeImage_1124707_30.stdout parsl.dataflow.dflow INFO: Standard error for task 172 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/5e46d950-66ac-4a6d-abe4-137e0d65fa9e_characterizeImage_1124707_30.stderr parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 767 try 0 failed parsl.dataflow.dflow INFO: Task 767 marked for retry parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/46d083a0-25ac-438f-ac7e-fa79f51f9dd8_calibrate_1124704_31.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/46d083a0-25ac-438f-ac7e-fa79f51f9dd8_calibrate_1124704_31.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 767 has memoization hash c715fbad0dfe0df5530a38b18d4a0e92 parsl.dataflow.memoization INFO: Task 767 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977eac0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 767 try 1 launched on executor multi with executor id 909 parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/46d083a0-25ac-438f-ac7e-fa79f51f9dd8_calibrate_1124704_31.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/46d083a0-25ac-438f-ac7e-fa79f51f9dd8_calibrate_1124704_31.stderr parsl.dataflow.dflow INFO: Task 257 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5d29ecb817ace1ef88f2e2abc13a8d3 with result from task 257 parsl.dataflow.dflow DEBUG: Task 257 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/624ec647-9674-4b6a-9eab-dbd66091a771_calibrate_1124707_16.stdout parsl.dataflow.dflow INFO: Standard error for task 257 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/624ec647-9674-4b6a-9eab-dbd66091a771_calibrate_1124707_16.stderr parsl.dataflow.dflow INFO: Task 496 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6de811eeda0f767c14f8594378b63898 with result from task 496 parsl.dataflow.dflow DEBUG: Task 496 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 497 has memoization hash c974fec96921108be1860e9f2e53e529 parsl.dataflow.memoization INFO: Task 497 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529851bc0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 497 try 0 launched on executor multi with executor id 910 parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e267f1fb-7711-45ef-995d-ea5876523e83_calibrate_1124707_40.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e267f1fb-7711-45ef-995d-ea5876523e83_calibrate_1124707_40.stderr parsl.dataflow.dflow INFO: Standard output for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/2de646cb-5a3b-4eb1-b42f-1fec41fbd3ef_characterizeImage_1124707_40.stdout parsl.dataflow.dflow INFO: Standard error for task 496 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/2de646cb-5a3b-4eb1-b42f-1fec41fbd3ef_characterizeImage_1124707_40.stderr parsl.dataflow.dflow INFO: Task 163 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd558371c3c59001669b77772899e7e9 with result from task 163 parsl.dataflow.dflow DEBUG: Task 163 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 164 has memoization hash 5a415a01f53634653abd16e273be9f7a parsl.dataflow.memoization INFO: Task 164 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352ad8fa60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 164 try 0 launched on executor multi with executor id 911 parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/12af302b-c2eb-4c8e-b809-e3a1b1b7ba3d_calibrate_1124704_37.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/12af302b-c2eb-4c8e-b809-e3a1b1b7ba3d_calibrate_1124704_37.stderr parsl.dataflow.dflow INFO: Standard output for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/36eddd68-e58d-40f4-acb2-120a34a6cd65_characterizeImage_1124704_37.stdout parsl.dataflow.dflow INFO: Standard error for task 163 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/36eddd68-e58d-40f4-acb2-120a34a6cd65_characterizeImage_1124704_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:00 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 647 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 46c4c7a341a10785dc67b17d4acb2eba with result from task 647 parsl.dataflow.dflow DEBUG: Task 647 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/91deba8e-51bb-44fc-b21f-d91b75d9876a_calibrate_1124710_1.stdout parsl.dataflow.dflow INFO: Standard error for task 647 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/91deba8e-51bb-44fc-b21f-d91b75d9876a_calibrate_1124710_1.stderr parsl.dataflow.dflow INFO: Task 652 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2b4accf1940013662856fce70dfcfd35 with result from task 652 parsl.dataflow.dflow DEBUG: Task 652 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 653 has memoization hash 7c5bf1bb6a5a1782b35fdf387ea1d207 parsl.dataflow.memoization INFO: Task 653 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296ebd80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 653 try 0 launched on executor multi with executor id 912 parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e781d8d5-84d5-4ce2-8c65-048d42c20d01_calibrate_1124707_55.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e781d8d5-84d5-4ce2-8c65-048d42c20d01_calibrate_1124707_55.stderr parsl.dataflow.dflow INFO: Standard output for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/d9e21934-7cee-428c-875c-c1f03aa9727b_characterizeImage_1124707_55.stdout parsl.dataflow.dflow INFO: Standard error for task 652 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/d9e21934-7cee-428c-875c-c1f03aa9727b_characterizeImage_1124707_55.stderr parsl.dataflow.dflow INFO: Task 842 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c463e4e15015838451350448e78d4e01 with result from task 842 parsl.dataflow.dflow DEBUG: Task 842 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/240aaa11-7a31-43f2-80bb-230ad0deae0c_calibrate_1124828_39.stdout parsl.dataflow.dflow INFO: Standard error for task 842 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/240aaa11-7a31-43f2-80bb-230ad0deae0c_calibrate_1124828_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:01 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 475 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 15fd7aaa28f7e7b42e16138f4144c9ef with result from task 475 parsl.dataflow.dflow DEBUG: Task 475 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 476 has memoization hash 81e52af518e524724e5e860c6f630b4b parsl.dataflow.memoization INFO: Task 476 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529851f80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 476 try 0 launched on executor multi with executor id 913 parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/8c30d298-aa18-4619-8089-f2bb3d31aebc_calibrate_1124707_11.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/8c30d298-aa18-4619-8089-f2bb3d31aebc_calibrate_1124707_11.stderr parsl.dataflow.dflow INFO: Standard output for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/38770e46-f90a-4e56-a26c-0792549e23bb_characterizeImage_1124707_11.stdout parsl.dataflow.dflow INFO: Standard error for task 475 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/38770e46-f90a-4e56-a26c-0792549e23bb_characterizeImage_1124707_11.stderr parsl.dataflow.dflow INFO: Task 431 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8aa3d0e025e0f80a395a03c88b1b4e23 with result from task 431 parsl.dataflow.dflow DEBUG: Task 431 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e9551fbe-b12d-480e-b3c0-79e129ef664f_calibrate_1124707_42.stdout parsl.dataflow.dflow INFO: Standard error for task 431 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e9551fbe-b12d-480e-b3c0-79e129ef664f_calibrate_1124707_42.stderr parsl.dataflow.dflow INFO: Task 518 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ef063c79aac55effdd6b3fa8e5cad70 with result from task 518 parsl.dataflow.dflow DEBUG: Task 518 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a484d460-9410-43f5-b221-d39ec845628a_calibrate_1124831_24.stdout parsl.dataflow.dflow INFO: Standard error for task 518 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/a484d460-9410-43f5-b221-d39ec845628a_calibrate_1124831_24.stderr parsl.dataflow.dflow INFO: Task 740 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cfca881c40258ad115791adf7ef76d50 with result from task 740 parsl.dataflow.dflow DEBUG: Task 740 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/77f774a1-76c4-470c-9944-906f4e2f1d28_calibrate_1124707_35.stdout parsl.dataflow.dflow INFO: Standard error for task 740 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/77f774a1-76c4-470c-9944-906f4e2f1d28_calibrate_1124707_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:02 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 917 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ba9b562ef4b620eef502338a76eff815 with result from task 917 parsl.dataflow.dflow DEBUG: Task 917 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/da210bcc-6a21-4771-adf1-348658d70a6a_calibrate_1124828_24.stdout parsl.dataflow.dflow INFO: Standard error for task 917 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/da210bcc-6a21-4771-adf1-348658d70a6a_calibrate_1124828_24.stderr parsl.dataflow.dflow INFO: Task 137 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d0cc177a685388813585f83a8931438 with result from task 137 parsl.dataflow.dflow DEBUG: Task 137 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d126533f-e0fd-4015-9bd1-579db1c41347_calibrate_1124707_36.stdout parsl.dataflow.dflow INFO: Standard error for task 137 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d126533f-e0fd-4015-9bd1-579db1c41347_calibrate_1124707_36.stderr parsl.dataflow.dflow INFO: Task 782 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c37594f51dcfbbfdc3d22ae2c4c31297 with result from task 782 parsl.dataflow.dflow DEBUG: Task 782 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/06e75b7a-ed34-4740-8d47-457583d1f484_calibrate_1124704_52.stdout parsl.dataflow.dflow INFO: Standard error for task 782 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/06e75b7a-ed34-4740-8d47-457583d1f484_calibrate_1124704_52.stderr parsl.dataflow.dflow INFO: Task 59 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0bda1c67ed2a36a7df4b0f22e0ef0ee6 with result from task 59 parsl.dataflow.dflow DEBUG: Task 59 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/a4f50922-e252-4473-b5c4-d269fa1a35d0_calibrate_1124710_30.stdout parsl.dataflow.dflow INFO: Standard error for task 59 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/a4f50922-e252-4473-b5c4-d269fa1a35d0_calibrate_1124710_30.stderr parsl.dataflow.dflow INFO: Task 202 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d94010c3f4976eca683f92b5998828db with result from task 202 parsl.dataflow.dflow DEBUG: Task 202 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 203 has memoization hash 52a1d1a680b139a5670d07d0e028e905 parsl.dataflow.memoization INFO: Task 203 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adae340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 203 try 0 launched on executor multi with executor id 914 parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/bed14d87-2dc3-4e58-aed3-1c8c799dbdea_calibrate_1124707_17.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/bed14d87-2dc3-4e58-aed3-1c8c799dbdea_calibrate_1124707_17.stderr parsl.dataflow.dflow INFO: Standard output for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/7f884313-ced4-44ab-9eb7-c27add222abe_characterizeImage_1124707_17.stdout parsl.dataflow.dflow INFO: Standard error for task 202 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/7f884313-ced4-44ab-9eb7-c27add222abe_characterizeImage_1124707_17.stderr parsl.dataflow.dflow INFO: Task 275 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a01f944200a0a54c1e7bcf1e9b15b074 with result from task 275 parsl.dataflow.dflow DEBUG: Task 275 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cb88bf09-d788-402d-9e36-f26b43697ff4_calibrate_1124707_3.stdout parsl.dataflow.dflow INFO: Standard error for task 275 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cb88bf09-d788-402d-9e36-f26b43697ff4_calibrate_1124707_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:03 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 923 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry eb74c148f5615fbada1e80da85486b80 with result from task 923 parsl.dataflow.dflow DEBUG: Task 923 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/5e53fbdc-d13a-4d27-b244-bab6e7f8d125_calibrate_1124831_48.stdout parsl.dataflow.dflow INFO: Standard error for task 923 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124831/5e53fbdc-d13a-4d27-b244-bab6e7f8d125_calibrate_1124831_48.stderr parsl.dataflow.dflow INFO: Task 581 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 84306f90dca26c5ccd1b417cd0d08ed5 with result from task 581 parsl.dataflow.dflow DEBUG: Task 581 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/74e5ecb2-076a-4834-8176-f92112501c09_calibrate_1124707_32.stdout parsl.dataflow.dflow INFO: Standard error for task 581 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/74e5ecb2-076a-4834-8176-f92112501c09_calibrate_1124707_32.stderr parsl.dataflow.dflow INFO: Task 65 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b68a719ca8fefebcdbe6fdfeff467372 with result from task 65 parsl.dataflow.dflow DEBUG: Task 65 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ddaa9ef4-fa3a-4a2a-93bb-3757464ab963_calibrate_1124704_9.stdout parsl.dataflow.dflow INFO: Standard error for task 65 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ddaa9ef4-fa3a-4a2a-93bb-3757464ab963_calibrate_1124704_9.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:04 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 347 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92b130887d8fd02a7b03736459d98572 with result from task 347 parsl.dataflow.dflow DEBUG: Task 347 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ed2c1157-8947-4245-8162-18de1cb44ba6_calibrate_1124710_27.stdout parsl.dataflow.dflow INFO: Standard error for task 347 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ed2c1157-8947-4245-8162-18de1cb44ba6_calibrate_1124710_27.stderr parsl.dataflow.dflow INFO: Task 401 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a205c45a778fa89a1b38f7aaed08e54a with result from task 401 parsl.dataflow.dflow DEBUG: Task 401 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/14b34b14-041a-4ae6-b4c9-4ef320975c8b_calibrate_1124704_46.stdout parsl.dataflow.dflow INFO: Standard error for task 401 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/14b34b14-041a-4ae6-b4c9-4ef320975c8b_calibrate_1124704_46.stderr parsl.dataflow.dflow INFO: Task 413 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3915334b5be25fe7a63ffd5a392f9e7d with result from task 413 parsl.dataflow.dflow DEBUG: Task 413 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cd208141-736b-430c-9b9d-62d04f85de97_calibrate_1124707_5.stdout parsl.dataflow.dflow INFO: Standard error for task 413 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cd208141-736b-430c-9b9d-62d04f85de97_calibrate_1124707_5.stderr parsl.dataflow.dflow INFO: Task 712 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3fa795a8bed149f19cd8c9c8de83af50 with result from task 712 parsl.dataflow.dflow DEBUG: Task 712 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 713 has memoization hash a8a205eeee6d170ed07c465f2aa08aff parsl.dataflow.memoization INFO: Task 713 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352973ad40> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 713 try 0 launched on executor multi with executor id 915 parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/c657083d-c8cc-4671-9619-3bd5f4283cda_calibrate_1124710_40.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/c657083d-c8cc-4671-9619-3bd5f4283cda_calibrate_1124710_40.stderr parsl.dataflow.dflow INFO: Standard output for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/79de0100-d2d0-4bc0-97ea-eafd4449ff6b_characterizeImage_1124710_40.stdout parsl.dataflow.dflow INFO: Standard error for task 712 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/79de0100-d2d0-4bc0-97ea-eafd4449ff6b_characterizeImage_1124710_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:05 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 181 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6d5efde68fa8dc0a121c022f2f54870a with result from task 181 parsl.dataflow.dflow DEBUG: Task 181 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 182 has memoization hash 2cecae0e9f2e62e3049f855d21ec845a parsl.dataflow.memoization INFO: Task 182 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adacd60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 182 try 0 launched on executor multi with executor id 916 parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/51e2f4cf-4670-4bd2-b532-5eb4b28f8529_calibrate_1124704_58.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/51e2f4cf-4670-4bd2-b532-5eb4b28f8529_calibrate_1124704_58.stderr parsl.dataflow.dflow INFO: Standard output for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/62080edc-8329-48e3-86ef-49935b381e07_characterizeImage_1124704_58.stdout parsl.dataflow.dflow INFO: Standard error for task 181 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/62080edc-8329-48e3-86ef-49935b381e07_characterizeImage_1124704_58.stderr parsl.dataflow.dflow INFO: Task 889 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea428097881235b9fb58b0460a44bde8 with result from task 889 parsl.dataflow.dflow DEBUG: Task 889 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 890 has memoization hash a8984f3b8ebee3e8e62db4b47f649831 parsl.dataflow.memoization INFO: Task 890 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960b560> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 890 try 0 launched on executor multi with executor id 917 parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/55c272be-89bf-4b21-bdbc-74a3f11deb26_calibrate_1124707_29.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/55c272be-89bf-4b21-bdbc-74a3f11deb26_calibrate_1124707_29.stderr parsl.dataflow.dflow INFO: Standard output for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8413d4a2-8de0-488f-bf68-7a748d99a6fe_characterizeImage_1124707_29.stdout parsl.dataflow.dflow INFO: Standard error for task 889 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/8413d4a2-8de0-488f-bf68-7a748d99a6fe_characterizeImage_1124707_29.stderr parsl.dataflow.dflow INFO: Task 907 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4ce8bc4ce82ef946192cce3472b00655 with result from task 907 parsl.dataflow.dflow DEBUG: Task 907 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 908 has memoization hash 32e1b8e6e62be72084363c885878face parsl.dataflow.memoization INFO: Task 908 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535296676a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 908 try 0 launched on executor multi with executor id 918 parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/59f61e14-e803-4cee-80ca-420c66a7160b_calibrate_1124707_50.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/59f61e14-e803-4cee-80ca-420c66a7160b_calibrate_1124707_50.stderr parsl.dataflow.dflow INFO: Standard output for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e1a23abc-5cb4-4ce5-a9cd-4dbf1d761404_characterizeImage_1124707_50.stdout parsl.dataflow.dflow INFO: Standard error for task 907 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e1a23abc-5cb4-4ce5-a9cd-4dbf1d761404_characterizeImage_1124707_50.stderr parsl.dataflow.dflow INFO: Task 98 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d16d8ed1a671348e14508aa75e52b52b with result from task 98 parsl.dataflow.dflow DEBUG: Task 98 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/94f6edee-d6f1-4008-87f0-8d3fe1e34e40_calibrate_1124704_14.stdout parsl.dataflow.dflow INFO: Standard error for task 98 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/94f6edee-d6f1-4008-87f0-8d3fe1e34e40_calibrate_1124704_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:06 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 451 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7da73a4e10559259f6a0dc1f92ff3d5f with result from task 451 parsl.dataflow.dflow DEBUG: Task 451 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 452 has memoization hash 41fbce7696e8b7f5f58d3f233ed9071c parsl.dataflow.memoization INFO: Task 452 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352981e840> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 452 try 0 launched on executor multi with executor id 919 parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/4ae27cd1-4a76-4a69-9e78-46bd15b4b07f_calibrate_1124710_60.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/4ae27cd1-4a76-4a69-9e78-46bd15b4b07f_calibrate_1124710_60.stderr parsl.dataflow.dflow INFO: Standard output for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/80096367-fa6c-4c17-90ac-046d6403c4c2_characterizeImage_1124710_60.stdout parsl.dataflow.dflow INFO: Standard error for task 451 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/80096367-fa6c-4c17-90ac-046d6403c4c2_characterizeImage_1124710_60.stderr parsl.dataflow.dflow INFO: Task 368 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9ab802d36b4911f7d1b76bc0e296f0ec with result from task 368 parsl.dataflow.dflow DEBUG: Task 368 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/317d451a-f2be-460c-8ce8-e24167358622_calibrate_1124710_26.stdout parsl.dataflow.dflow INFO: Standard error for task 368 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/317d451a-f2be-460c-8ce8-e24167358622_calibrate_1124710_26.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:07 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 218 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7438f370f14044e1897a5b65ac7e93ac with result from task 218 parsl.dataflow.dflow DEBUG: Task 218 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/8d6c444a-e44f-43cc-84a7-a3b11db1afc6_calibrate_1124710_10.stdout parsl.dataflow.dflow INFO: Standard error for task 218 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/8d6c444a-e44f-43cc-84a7-a3b11db1afc6_calibrate_1124710_10.stderr parsl.dataflow.dflow INFO: Task 545 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 493fd56eee42e3c70663a0eb70a2dee3 with result from task 545 parsl.dataflow.dflow DEBUG: Task 545 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ab3f8d26-8aa8-4f73-8b7e-b897ce496856_calibrate_1124710_29.stdout parsl.dataflow.dflow INFO: Standard error for task 545 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ab3f8d26-8aa8-4f73-8b7e-b897ce496856_calibrate_1124710_29.stderr parsl.dataflow.dflow INFO: Task 611 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6daee7f891bc6d0cf37cd95ddc7625c4 with result from task 611 parsl.dataflow.dflow DEBUG: Task 611 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ca919ccc-75aa-44fb-b461-fbd2aa28ef75_calibrate_1124707_18.stdout parsl.dataflow.dflow INFO: Standard error for task 611 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ca919ccc-75aa-44fb-b461-fbd2aa28ef75_calibrate_1124707_18.stderr parsl.dataflow.dflow INFO: Task 53 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb0b2d8bd1cdbc8cd3626da2cb7053a5 with result from task 53 parsl.dataflow.dflow DEBUG: Task 53 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cc39350e-05c5-4b1e-ad7f-a2cd82e68861_calibrate_1124707_41.stdout parsl.dataflow.dflow INFO: Standard error for task 53 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cc39350e-05c5-4b1e-ad7f-a2cd82e68861_calibrate_1124707_41.stderr parsl.dataflow.dflow INFO: Task 895 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3a33da76d4a48aeb4d0095884081e502 with result from task 895 parsl.dataflow.dflow DEBUG: Task 895 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 896 has memoization hash 4feb3ca7dab65154040d1fbdfee23ac1 parsl.dataflow.memoization INFO: Task 896 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960bba0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 896 try 0 launched on executor multi with executor id 920 parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/20927db7-5ea3-49cc-8f82-fd6209a2a39a_calibrate_1124710_59.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/20927db7-5ea3-49cc-8f82-fd6209a2a39a_calibrate_1124710_59.stderr parsl.dataflow.dflow INFO: Standard output for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d1071e99-4e87-48c9-b246-90a10057faaf_characterizeImage_1124710_59.stdout parsl.dataflow.dflow INFO: Standard error for task 895 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/d1071e99-4e87-48c9-b246-90a10057faaf_characterizeImage_1124710_59.stderr parsl.dataflow.dflow INFO: Task 272 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34d86fff3f2c6460006b20021c8b2dcb with result from task 272 parsl.dataflow.dflow DEBUG: Task 272 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1bd5ad9c-532b-45cd-8039-44da2e8d5eee_calibrate_1124707_47.stdout parsl.dataflow.dflow INFO: Standard error for task 272 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1bd5ad9c-532b-45cd-8039-44da2e8d5eee_calibrate_1124707_47.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:08 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 260 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0c83fcc8f09aad46676ba1597a077bf7 with result from task 260 parsl.dataflow.dflow DEBUG: Task 260 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f4fba208-e15e-4bc6-a34b-a41a9e71105c_calibrate_1124828_27.stdout parsl.dataflow.dflow INFO: Standard error for task 260 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124828/f4fba208-e15e-4bc6-a34b-a41a9e71105c_calibrate_1124828_27.stderr parsl.dataflow.dflow INFO: Task 656 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 794710c5bfbea690046d1a0751fb7904 with result from task 656 parsl.dataflow.dflow DEBUG: Task 656 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/0bda3662-abb4-4e14-a7e2-b163739a86e0_calibrate_1124710_19.stdout parsl.dataflow.dflow INFO: Standard error for task 656 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/0bda3662-abb4-4e14-a7e2-b163739a86e0_calibrate_1124710_19.stderr parsl.dataflow.dflow INFO: Task 641 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 457a009ba6af073990f6fb6894042cc0 with result from task 641 parsl.dataflow.dflow DEBUG: Task 641 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/35607b2a-b901-4a12-9e3c-b0bb67ee8a18_calibrate_1124707_56.stdout parsl.dataflow.dflow INFO: Standard error for task 641 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/35607b2a-b901-4a12-9e3c-b0bb67ee8a18_calibrate_1124707_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:09 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 443 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d96c659a4cd5614bc30c354d8b294506 with result from task 443 parsl.dataflow.dflow DEBUG: Task 443 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/0c906e6c-3a9e-49a2-9042-a96552ed3d60_calibrate_1124704_10.stdout parsl.dataflow.dflow INFO: Standard error for task 443 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/0c906e6c-3a9e-49a2-9042-a96552ed3d60_calibrate_1124704_10.stderr parsl.dataflow.dflow INFO: Task 547 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2040c3158e110b96c14e00df4a68f9c1 with result from task 547 parsl.dataflow.dflow DEBUG: Task 547 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 548 has memoization hash 6e0264845d5d0abe388b9aa9f638304b parsl.dataflow.memoization INFO: Task 548 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352969c5e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 548 try 0 launched on executor multi with executor id 921 parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/12f479a3-9f88-4190-b4b2-25b0f77d967d_calibrate_1124710_4.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/12f479a3-9f88-4190-b4b2-25b0f77d967d_calibrate_1124710_4.stderr parsl.dataflow.dflow INFO: Standard output for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/45df5da3-3831-4f4a-9681-7a99eb61dc4f_characterizeImage_1124710_4.stdout parsl.dataflow.dflow INFO: Standard error for task 547 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/45df5da3-3831-4f4a-9681-7a99eb61dc4f_characterizeImage_1124710_4.stderr parsl.dataflow.dflow INFO: Task 383 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 744c9c3c7e889843fb6a2f870eb60962 with result from task 383 parsl.dataflow.dflow DEBUG: Task 383 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/a81eb202-93ef-462d-9721-06f80500079b_calibrate_1124710_51.stdout parsl.dataflow.dflow INFO: Standard error for task 383 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/a81eb202-93ef-462d-9721-06f80500079b_calibrate_1124710_51.stderr parsl.dataflow.dflow INFO: Task 626 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b858bed7d8a8e170d6b6db338d261a12 with result from task 626 parsl.dataflow.dflow DEBUG: Task 626 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/8103d01c-dd47-4fb0-b816-1aebdf2f2d05_calibrate_1124710_45.stdout parsl.dataflow.dflow INFO: Standard error for task 626 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/8103d01c-dd47-4fb0-b816-1aebdf2f2d05_calibrate_1124710_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:10 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 668 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 01aa482d5bd573e4fbd6d8889c32cb8c with result from task 668 parsl.dataflow.dflow DEBUG: Task 668 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/e37aaa39-5822-41c3-a95d-aa28b1a70791_calibrate_1124704_62.stdout parsl.dataflow.dflow INFO: Standard error for task 668 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/e37aaa39-5822-41c3-a95d-aa28b1a70791_calibrate_1124704_62.stderr parsl.dataflow.dflow INFO: Task 725 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5c9f71326ebfe5c278348db2b47e659c with result from task 725 parsl.dataflow.dflow DEBUG: Task 725 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/1c57f535-10e1-46aa-8a93-93e2dd9a7820_calibrate_1124704_5.stdout parsl.dataflow.dflow INFO: Standard error for task 725 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/1c57f535-10e1-46aa-8a93-93e2dd9a7820_calibrate_1124704_5.stderr parsl.dataflow.dflow INFO: Task 311 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da398009589aa431749456b122c200da with result from task 311 parsl.dataflow.dflow DEBUG: Task 311 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/862f4de7-835d-48ba-881a-a4e8a993c72d_calibrate_1124704_39.stdout parsl.dataflow.dflow INFO: Standard error for task 311 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/862f4de7-835d-48ba-881a-a4e8a993c72d_calibrate_1124704_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:11 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 230 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 829aa4d6afe9ad6174eb426e9c877cca with result from task 230 parsl.dataflow.dflow DEBUG: Task 230 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7ca095b8-4393-4df9-b529-12f8f355620d_calibrate_1124710_50.stdout parsl.dataflow.dflow INFO: Standard error for task 230 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7ca095b8-4393-4df9-b529-12f8f355620d_calibrate_1124710_50.stderr parsl.dataflow.dflow INFO: Task 365 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry de4116e7bd0e5e7b33470aec12d5c89d with result from task 365 parsl.dataflow.dflow DEBUG: Task 365 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/f5222419-4921-4112-a25b-47e7ff3fe571_calibrate_1124704_19.stdout parsl.dataflow.dflow INFO: Standard error for task 365 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/f5222419-4921-4112-a25b-47e7ff3fe571_calibrate_1124704_19.stderr parsl.dataflow.dflow INFO: Task 113 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c946a2dcefc54baf10d60e5501b95b46 with result from task 113 parsl.dataflow.dflow DEBUG: Task 113 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ea75c6e3-82ba-47a6-a509-54ee2af60caa_calibrate_1124704_49.stdout parsl.dataflow.dflow INFO: Standard error for task 113 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ea75c6e3-82ba-47a6-a509-54ee2af60caa_calibrate_1124704_49.stderr parsl.dataflow.dflow INFO: Task 680 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4cb6230821350dc7c9f9451618364e2b with result from task 680 parsl.dataflow.dflow DEBUG: Task 680 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/152cabc0-8fcb-41f6-9a2c-2f69097d5b8d_calibrate_1124710_3.stdout parsl.dataflow.dflow INFO: Standard error for task 680 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/152cabc0-8fcb-41f6-9a2c-2f69097d5b8d_calibrate_1124710_3.stderr parsl.dataflow.dflow INFO: Task 89 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d0776adf3864b657d3130fea6099dde0 with result from task 89 parsl.dataflow.dflow DEBUG: Task 89 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/6a58f80e-b0f2-41d5-9be7-3d05699fd27c_calibrate_1124710_41.stdout parsl.dataflow.dflow INFO: Standard error for task 89 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/6a58f80e-b0f2-41d5-9be7-3d05699fd27c_calibrate_1124710_41.stderr parsl.dataflow.dflow INFO: Task 527 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7dd12fa8f62c12727f3b0ed1ab5ac876 with result from task 527 parsl.dataflow.dflow DEBUG: Task 527 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/668bc0fe-6487-4c53-96de-b296cb3ad825_calibrate_1124710_42.stdout parsl.dataflow.dflow INFO: Standard error for task 527 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/668bc0fe-6487-4c53-96de-b296cb3ad825_calibrate_1124710_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:11 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 590 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 90c10adc6442d42db0eabc56fdb4c8fe with result from task 590 parsl.dataflow.dflow DEBUG: Task 590 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/347633b5-5ebf-48ea-892b-33e832aa2f88_calibrate_1124704_35.stdout parsl.dataflow.dflow INFO: Standard error for task 590 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/347633b5-5ebf-48ea-892b-33e832aa2f88_calibrate_1124704_35.stderr parsl.dataflow.dflow INFO: Task 326 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5491a03ba50c7f32bc06610bd4ec6ea8 with result from task 326 parsl.dataflow.dflow DEBUG: Task 326 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/9e1c70a0-c52b-4cb0-b9b9-e999fb82ebee_calibrate_1124710_15.stdout parsl.dataflow.dflow INFO: Standard error for task 326 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/9e1c70a0-c52b-4cb0-b9b9-e999fb82ebee_calibrate_1124710_15.stderr parsl.dataflow.dflow INFO: Task 623 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 07d0676790eaa560905dda61f8d09cf1 with result from task 623 parsl.dataflow.dflow DEBUG: Task 623 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/a02cdab4-8cb2-40e4-b547-2720976a55f8_calibrate_1124704_29.stdout parsl.dataflow.dflow INFO: Standard error for task 623 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/a02cdab4-8cb2-40e4-b547-2720976a55f8_calibrate_1124704_29.stderr parsl.dataflow.dflow INFO: Task 638 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 584b6e426300c975662d27b6b6b47d8a with result from task 638 parsl.dataflow.dflow DEBUG: Task 638 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d5d89f0d-2576-431c-9add-995eb3f58800_calibrate_1124707_19.stdout parsl.dataflow.dflow INFO: Standard error for task 638 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d5d89f0d-2576-431c-9add-995eb3f58800_calibrate_1124707_19.stderr parsl.dataflow.dflow INFO: Task 533 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f4033a874525512e131f3102f4e8791d with result from task 533 parsl.dataflow.dflow DEBUG: Task 533 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/fec92767-c40f-4bcf-bbc1-ec6010be1b72_calibrate_1124710_35.stdout parsl.dataflow.dflow INFO: Standard error for task 533 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/fec92767-c40f-4bcf-bbc1-ec6010be1b72_calibrate_1124710_35.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:12 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 683 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 065d532909913e592ab9a41d8398e052 with result from task 683 parsl.dataflow.dflow DEBUG: Task 683 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1393cf79-ae33-40c7-8273-da7f81407f67_calibrate_1124710_11.stdout parsl.dataflow.dflow INFO: Standard error for task 683 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1393cf79-ae33-40c7-8273-da7f81407f67_calibrate_1124710_11.stderr parsl.dataflow.dflow INFO: Task 563 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 68f36f9a52c57246cde62a25d68cf93a with result from task 563 parsl.dataflow.dflow DEBUG: Task 563 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/96c86f86-aeb2-488d-9514-9528331ad979_calibrate_1124707_26.stdout parsl.dataflow.dflow INFO: Standard error for task 563 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/96c86f86-aeb2-488d-9514-9528331ad979_calibrate_1124707_26.stderr parsl.dataflow.dflow INFO: Task 296 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 12da49202063734aa45b018478a41172 with result from task 296 parsl.dataflow.dflow DEBUG: Task 296 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1b95afe3-989b-4072-9966-4a0fb0c2a4e9_calibrate_1124707_46.stdout parsl.dataflow.dflow INFO: Standard error for task 296 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1b95afe3-989b-4072-9966-4a0fb0c2a4e9_calibrate_1124707_46.stderr parsl.dataflow.dflow INFO: Task 644 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8f205bff390192006ba130def9d3ea46 with result from task 644 parsl.dataflow.dflow DEBUG: Task 644 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/768ecffc-261b-4ebc-b8cd-3cc11c642d0a_calibrate_1124707_6.stdout parsl.dataflow.dflow INFO: Standard error for task 644 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/768ecffc-261b-4ebc-b8cd-3cc11c642d0a_calibrate_1124707_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:13 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 542 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d81d1380ae0e7392a0408f36c45bcb65 with result from task 542 parsl.dataflow.dflow DEBUG: Task 542 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/0b01d29a-fcf3-4366-ba93-53d2418305db_calibrate_1124704_33.stdout parsl.dataflow.dflow INFO: Standard error for task 542 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/0b01d29a-fcf3-4366-ba93-53d2418305db_calibrate_1124704_33.stderr parsl.dataflow.dflow INFO: Task 122 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ae21e0981a4240ef8d3e1110d03177f with result from task 122 parsl.dataflow.dflow DEBUG: Task 122 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/65580c76-b944-40cd-9dbe-aa894d5ab01a_calibrate_1124704_50.stdout parsl.dataflow.dflow INFO: Standard error for task 122 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/65580c76-b944-40cd-9dbe-aa894d5ab01a_calibrate_1124704_50.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:14 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 215 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5fbc0fefaeaaec8c5fcbee3506242917 with result from task 215 parsl.dataflow.dflow DEBUG: Task 215 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/b970717d-091a-4d55-8f3a-b5690ca6ac6a_calibrate_1124704_34.stdout parsl.dataflow.dflow INFO: Standard error for task 215 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/b970717d-091a-4d55-8f3a-b5690ca6ac6a_calibrate_1124704_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:14 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 224 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 29527bedce098781a85b2dd2346a0e74 with result from task 224 parsl.dataflow.dflow DEBUG: Task 224 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/b1f9002e-672d-44de-993b-98c39a333ce0_calibrate_1124704_56.stdout parsl.dataflow.dflow INFO: Standard error for task 224 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/b1f9002e-672d-44de-993b-98c39a333ce0_calibrate_1124704_56.stderr parsl.dataflow.dflow INFO: Task 344 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94bcdf27701a6ed9be3908e2bb6fec0d with result from task 344 parsl.dataflow.dflow DEBUG: Task 344 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/295e2d70-ee07-482f-9da6-93f3ef09bfbe_calibrate_1124707_38.stdout parsl.dataflow.dflow INFO: Standard error for task 344 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/295e2d70-ee07-482f-9da6-93f3ef09bfbe_calibrate_1124707_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:15 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 143 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c5f36ac44f44e3dde3d3169cd86201e2 with result from task 143 parsl.dataflow.dflow DEBUG: Task 143 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1e902908-2c2f-4de9-ac26-827a81507929_calibrate_1124710_32.stdout parsl.dataflow.dflow INFO: Standard error for task 143 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1e902908-2c2f-4de9-ac26-827a81507929_calibrate_1124710_32.stderr parsl.dataflow.dflow INFO: Task 131 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8be55351e218efcc620404de725d5550 with result from task 131 parsl.dataflow.dflow DEBUG: Task 131 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/9a84967e-3ccc-44cb-9af2-e7725d673076_calibrate_1124704_22.stdout parsl.dataflow.dflow INFO: Standard error for task 131 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/9a84967e-3ccc-44cb-9af2-e7725d673076_calibrate_1124704_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:15 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:16 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:16 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 665 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c78d426756a518d01c53a6dd76c21f4 with result from task 665 parsl.dataflow.dflow DEBUG: Task 665 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/aa713c1b-1853-44a0-bb76-ffd5b9972b76_calibrate_1124710_39.stdout parsl.dataflow.dflow INFO: Standard error for task 665 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/aa713c1b-1853-44a0-bb76-ffd5b9972b76_calibrate_1124710_39.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:16 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 425 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 44482ec22626935f6ddc46f01a72bd09 with result from task 425 parsl.dataflow.dflow DEBUG: Task 425 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/bea6868c-159a-44fb-908e-24a63c00d5e1_calibrate_1124704_41.stdout parsl.dataflow.dflow INFO: Standard error for task 425 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/bea6868c-159a-44fb-908e-24a63c00d5e1_calibrate_1124704_41.stderr parsl.dataflow.dflow INFO: Task 233 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 38f7c2a4cbc6952d8547d46b32655233 with result from task 233 parsl.dataflow.dflow DEBUG: Task 233 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/d61171d0-282a-4567-9f9d-bda7d46583b8_calibrate_1124710_6.stdout parsl.dataflow.dflow INFO: Standard error for task 233 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/d61171d0-282a-4567-9f9d-bda7d46583b8_calibrate_1124710_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:17 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 419 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 777fcc099c244ca8e79a5b0588037f26 with result from task 419 parsl.dataflow.dflow DEBUG: Task 419 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/953cee98-635b-4b88-b0b0-c03db25c2c00_calibrate_1124710_53.stdout parsl.dataflow.dflow INFO: Standard error for task 419 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/953cee98-635b-4b88-b0b0-c03db25c2c00_calibrate_1124710_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:17 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 242 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a36ca329d0cc44c6f1c625f28bffc7c3 with result from task 242 parsl.dataflow.dflow DEBUG: Task 242 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/9b583028-6d7a-43fa-be1b-82939ada8db0_calibrate_1124704_8.stdout parsl.dataflow.dflow INFO: Standard error for task 242 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/9b583028-6d7a-43fa-be1b-82939ada8db0_calibrate_1124704_8.stderr parsl.dataflow.dflow INFO: Task 836 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5db2c4048fa4e0878bb5af706a5951ff with result from task 836 parsl.dataflow.dflow DEBUG: Task 836 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ced8c61e-52c4-456a-b7a3-a4a24c62ab7f_calibrate_1124707_57.stdout parsl.dataflow.dflow INFO: Standard error for task 836 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ced8c61e-52c4-456a-b7a3-a4a24c62ab7f_calibrate_1124707_57.stderr parsl.dataflow.dflow INFO: Task 353 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0cf6adb35172671d13f100e6c816cec with result from task 353 parsl.dataflow.dflow DEBUG: Task 353 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/423bf96a-6347-4665-902f-74613fd47d6b_calibrate_1124704_4.stdout parsl.dataflow.dflow INFO: Standard error for task 353 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/423bf96a-6347-4665-902f-74613fd47d6b_calibrate_1124704_4.stderr parsl.dataflow.dflow INFO: Task 887 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c56454f6b4b859893ed93f226b3c2afb with result from task 887 parsl.dataflow.dflow DEBUG: Task 887 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/dc676de6-2ea3-4e6a-9a9d-8994d9330848_calibrate_1124704_57.stdout parsl.dataflow.dflow INFO: Standard error for task 887 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/dc676de6-2ea3-4e6a-9a9d-8994d9330848_calibrate_1124704_57.stderr parsl.dataflow.dflow INFO: Task 799 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33cf5a493c7bf01b294fe90a72f2271d with result from task 799 parsl.dataflow.dflow DEBUG: Task 799 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 800 has memoization hash c0045da3f8752bde0a1ec0b898794d4e parsl.dataflow.memoization INFO: Task 800 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295cbf60> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 800 try 0 launched on executor multi with executor id 922 parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7ab9af8c-7f53-466f-94ff-a5243e5b190d_calibrate_1124710_7.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7ab9af8c-7f53-466f-94ff-a5243e5b190d_calibrate_1124710_7.stderr parsl.dataflow.dflow INFO: Standard output for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/03720eed-cdeb-4809-8082-30aa1fdaac84_characterizeImage_1124710_7.stdout parsl.dataflow.dflow INFO: Standard error for task 799 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/03720eed-cdeb-4809-8082-30aa1fdaac84_characterizeImage_1124710_7.stderr parsl.dataflow.dflow INFO: Task 596 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8e16394e51b97ba76da0fffecf4b50b with result from task 596 parsl.dataflow.dflow DEBUG: Task 596 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/db265463-0900-41f6-b433-65962dad4b0e_calibrate_1124707_58.stdout parsl.dataflow.dflow INFO: Standard error for task 596 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/db265463-0900-41f6-b433-65962dad4b0e_calibrate_1124707_58.stderr parsl.dataflow.dflow INFO: Task 599 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3f027e8cce3fa5d534021554b9cf2918 with result from task 599 parsl.dataflow.dflow DEBUG: Task 599 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8f1b9710-be69-47f2-8561-e219418846ec_calibrate_1124704_1.stdout parsl.dataflow.dflow INFO: Standard error for task 599 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8f1b9710-be69-47f2-8561-e219418846ec_calibrate_1124704_1.stderr parsl.dataflow.dflow INFO: Task 617 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 95ba8342b3ad70fffa8f932c327c0eec with result from task 617 parsl.dataflow.dflow DEBUG: Task 617 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/034d316f-3e40-444d-b3d2-3280c22b756e_calibrate_1124710_58.stdout parsl.dataflow.dflow INFO: Standard error for task 617 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/034d316f-3e40-444d-b3d2-3280c22b756e_calibrate_1124710_58.stderr parsl.dataflow.dflow INFO: Task 239 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 34ea410ba8141b401779d65cac8cdeb9 with result from task 239 parsl.dataflow.dflow DEBUG: Task 239 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1aba9bad-8c9d-456b-b3f2-1f4cab88b235_calibrate_1124710_17.stdout parsl.dataflow.dflow INFO: Standard error for task 239 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1aba9bad-8c9d-456b-b3f2-1f4cab88b235_calibrate_1124710_17.stderr parsl.dataflow.dflow INFO: Task 404 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3472590723ddb31f0029576a01a5dd10 with result from task 404 parsl.dataflow.dflow DEBUG: Task 404 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/f1a978f4-a814-41d2-aac1-566216e00455_calibrate_1124707_15.stdout parsl.dataflow.dflow INFO: Standard error for task 404 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/f1a978f4-a814-41d2-aac1-566216e00455_calibrate_1124707_15.stderr parsl.dataflow.dflow INFO: Task 169 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 53f8cd51843947d9fe05b8968f453b70 with result from task 169 parsl.dataflow.dflow DEBUG: Task 169 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 170 has memoization hash f7fe7cd42245389c46994386eb26bb18 parsl.dataflow.memoization INFO: Task 170 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352adac0e0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 170 try 0 launched on executor multi with executor id 923 parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8b83b24a-0c75-48be-b1e1-eb0d459d3853_calibrate_1124704_25.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8b83b24a-0c75-48be-b1e1-eb0d459d3853_calibrate_1124704_25.stderr parsl.dataflow.dflow INFO: Standard output for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f7e04282-b7ea-4b73-a36c-9738bee66f45_characterizeImage_1124704_25.stdout parsl.dataflow.dflow INFO: Standard error for task 169 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/f7e04282-b7ea-4b73-a36c-9738bee66f45_characterizeImage_1124704_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:19 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:20 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 791 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8443d45b5b83f27932bbc8b52e8ccfa6 with result from task 791 parsl.dataflow.dflow DEBUG: Task 791 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/58000cf4-f5c1-4a2a-b9e5-75ae9042c03e_calibrate_1124704_47.stdout parsl.dataflow.dflow INFO: Standard error for task 791 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/58000cf4-f5c1-4a2a-b9e5-75ae9042c03e_calibrate_1124704_47.stderr parsl.dataflow.dflow INFO: Task 557 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0a742cfc9fee604c552809d39de3d6c8 with result from task 557 parsl.dataflow.dflow DEBUG: Task 557 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/a3637201-6b78-40b8-9b56-cf4d8706c45c_calibrate_1124704_17.stdout parsl.dataflow.dflow INFO: Standard error for task 557 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/a3637201-6b78-40b8-9b56-cf4d8706c45c_calibrate_1124704_17.stderr parsl.dataflow.dflow INFO: Task 614 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a7a07175145d1a60fd321d3679042495 with result from task 614 parsl.dataflow.dflow DEBUG: Task 614 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/3a9171dd-200e-4199-b878-cefa50df8ff3_calibrate_1124710_24.stdout parsl.dataflow.dflow INFO: Standard error for task 614 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/3a9171dd-200e-4199-b878-cefa50df8ff3_calibrate_1124710_24.stderr parsl.dataflow.dflow INFO: Task 919 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2715dc843bb514ca10f76f5df9082302 with result from task 919 parsl.dataflow.dflow DEBUG: Task 919 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 920 has memoization hash e662d26ec432b5ba2301844b4f38d51e parsl.dataflow.memoization INFO: Task 920 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529664b80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 920 try 0 launched on executor multi with executor id 924 parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/a82a9349-d637-4143-824c-78ed263b7c73_calibrate_1124704_48.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/a82a9349-d637-4143-824c-78ed263b7c73_calibrate_1124704_48.stderr parsl.dataflow.dflow INFO: Standard output for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/ae27b3bf-54fa-4f3b-a843-839ce9c4fb01_characterizeImage_1124704_48.stdout parsl.dataflow.dflow INFO: Standard error for task 919 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/ae27b3bf-54fa-4f3b-a843-839ce9c4fb01_characterizeImage_1124704_48.stderr parsl.dataflow.dflow INFO: Task 866 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8e46dab2d052b4663e42837b9edeee1e with result from task 866 parsl.dataflow.dflow DEBUG: Task 866 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1528c911-116f-40f6-b486-b5e3f627c5d6_calibrate_1124707_14.stdout parsl.dataflow.dflow INFO: Standard error for task 866 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1528c911-116f-40f6-b486-b5e3f627c5d6_calibrate_1124707_14.stderr parsl.dataflow.dflow INFO: Task 302 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 566ae98cdb1dcf16487d9abbd8177243 with result from task 302 parsl.dataflow.dflow DEBUG: Task 302 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ad07d3a0-20a4-4495-9922-4d3978901d74_calibrate_1124707_43.stdout parsl.dataflow.dflow INFO: Standard error for task 302 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ad07d3a0-20a4-4495-9922-4d3978901d74_calibrate_1124707_43.stderr parsl.dataflow.dflow INFO: Task 277 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2c6533e2bd8072c8535d0fc90d16c197 with result from task 277 parsl.dataflow.dflow DEBUG: Task 277 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 278 has memoization hash 1e30062eeb161e72a65902ab846b05d0 parsl.dataflow.memoization INFO: Task 278 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297ab1a0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 278 try 0 launched on executor multi with executor id 925 parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1ba10e4c-6a44-44d9-9aa9-4a2678062525_calibrate_1124707_24.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1ba10e4c-6a44-44d9-9aa9-4a2678062525_calibrate_1124707_24.stderr parsl.dataflow.dflow INFO: Standard output for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/1ca09767-8beb-451f-b05d-15452299f4e9_characterizeImage_1124707_24.stdout parsl.dataflow.dflow INFO: Standard error for task 277 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/1ca09767-8beb-451f-b05d-15452299f4e9_characterizeImage_1124707_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:21 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 380 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 51cc15169b39d8a2232564c47af8daf2 with result from task 380 parsl.dataflow.dflow DEBUG: Task 380 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ebdd4b7c-08d0-43ef-b1b1-6c9f71c656df_calibrate_1124704_45.stdout parsl.dataflow.dflow INFO: Standard error for task 380 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ebdd4b7c-08d0-43ef-b1b1-6c9f71c656df_calibrate_1124704_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:22 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 605 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 23252e7a9823b8e845f92422ff7b36da with result from task 605 parsl.dataflow.dflow DEBUG: Task 605 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/249ef6ab-9a9a-4bae-a9ad-8e361cb51ec4_calibrate_1124710_54.stdout parsl.dataflow.dflow INFO: Standard error for task 605 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/249ef6ab-9a9a-4bae-a9ad-8e361cb51ec4_calibrate_1124710_54.stderr parsl.dataflow.dflow INFO: Task 757 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7529792690d62d7f4b00e51fefca08b4 with result from task 757 parsl.dataflow.dflow DEBUG: Task 757 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 758 has memoization hash b58e169f260f94b7ae05363a9ef95543 parsl.dataflow.memoization INFO: Task 758 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977e340> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 758 try 0 launched on executor multi with executor id 926 parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cbd6f5c1-f71c-4d37-93ee-d5a48bdd33c9_calibrate_1124707_7.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cbd6f5c1-f71c-4d37-93ee-d5a48bdd33c9_calibrate_1124707_7.stderr parsl.dataflow.dflow INFO: Standard output for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/38d8d024-8186-409f-8a89-25bd4a9d97de_characterizeImage_1124707_7.stdout parsl.dataflow.dflow INFO: Standard error for task 757 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/38d8d024-8186-409f-8a89-25bd4a9d97de_characterizeImage_1124707_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:22 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:22 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 536 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3e1963aa48ec7f4d76c38f8861721e7a with result from task 536 parsl.dataflow.dflow DEBUG: Task 536 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/28c36749-9191-4bfb-90de-3edf776c1d76_calibrate_1124707_12.stdout parsl.dataflow.dflow INFO: Standard error for task 536 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/28c36749-9191-4bfb-90de-3edf776c1d76_calibrate_1124707_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:23 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 857 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7e518ecaa0faa2993c2fe1fa476df1bc with result from task 857 parsl.dataflow.dflow DEBUG: Task 857 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/3d632c1a-13e8-4444-a3c3-f74f10a8282a_calibrate_1124710_5.stdout parsl.dataflow.dflow INFO: Standard error for task 857 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/3d632c1a-13e8-4444-a3c3-f74f10a8282a_calibrate_1124710_5.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:23 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 416 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bb21dea1fc3efa82ea8fa99a74f0ff72 with result from task 416 parsl.dataflow.dflow DEBUG: Task 416 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/29cbee83-4aae-469d-a1bd-6a1045bc2637_calibrate_1124707_9.stdout parsl.dataflow.dflow INFO: Standard error for task 416 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/29cbee83-4aae-469d-a1bd-6a1045bc2637_calibrate_1124707_9.stderr parsl.dataflow.dflow INFO: Task 686 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cdf7fcf2652071fad348d995bebc35f0 with result from task 686 parsl.dataflow.dflow DEBUG: Task 686 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/f55cbeed-aa88-43d6-b1dc-e6687d31e5fd_calibrate_1124707_4.stdout parsl.dataflow.dflow INFO: Standard error for task 686 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/f55cbeed-aa88-43d6-b1dc-e6687d31e5fd_calibrate_1124707_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:23 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:24 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 551 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1ca1cd6ab9e015f717cf956d08ce91c1 with result from task 551 parsl.dataflow.dflow DEBUG: Task 551 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/5f052231-aee6-49bd-b4e6-d0a4e08aef64_calibrate_1124704_53.stdout parsl.dataflow.dflow INFO: Standard error for task 551 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/5f052231-aee6-49bd-b4e6-d0a4e08aef64_calibrate_1124704_53.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:24 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 698 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb4c321c8c9a21beb3e74d5aa7eafd79 with result from task 698 parsl.dataflow.dflow DEBUG: Task 698 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/b2ac927f-5824-491c-92f6-9c1f827e008a_calibrate_1124710_36.stdout parsl.dataflow.dflow INFO: Standard error for task 698 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/b2ac927f-5824-491c-92f6-9c1f827e008a_calibrate_1124710_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:24 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 659 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5144703371afbfacad41091eaafc58a0 with result from task 659 parsl.dataflow.dflow DEBUG: Task 659 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/4eaf136c-c348-4fd8-bebc-330e92cc1512_calibrate_1124704_30.stdout parsl.dataflow.dflow INFO: Standard error for task 659 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/4eaf136c-c348-4fd8-bebc-330e92cc1512_calibrate_1124704_30.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:25 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 587 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 98b8dde43cb7fb4a2c2551479652ae7c with result from task 587 parsl.dataflow.dflow DEBUG: Task 587 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/a4b922bc-feab-4e36-9698-5d01c542b3ad_calibrate_1124707_21.stdout parsl.dataflow.dflow INFO: Standard error for task 587 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/a4b922bc-feab-4e36-9698-5d01c542b3ad_calibrate_1124707_21.stderr parsl.dataflow.dflow INFO: Task 929 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e4e81c19280a94b50f7c0ec4e4e7dbfb with result from task 929 parsl.dataflow.dflow DEBUG: Task 929 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/4e0abbbc-d63e-47ad-b849-b3809b3e9ce9_calibrate_1124704_36.stdout parsl.dataflow.dflow INFO: Standard error for task 929 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/4e0abbbc-d63e-47ad-b849-b3809b3e9ce9_calibrate_1124704_36.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:25 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 833 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b5d8ce2d92e392c916c4aa33e4d240a9 with result from task 833 parsl.dataflow.dflow DEBUG: Task 833 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ec455e3f-4689-4b40-9f42-3f1a3d648a2e_calibrate_1124704_13.stdout parsl.dataflow.dflow INFO: Standard error for task 833 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ec455e3f-4689-4b40-9f42-3f1a3d648a2e_calibrate_1124704_13.stderr parsl.dataflow.dflow INFO: Task 44 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f96ed19427896a6032cb6dbc1b5e20b9 with result from task 44 parsl.dataflow.dflow DEBUG: Task 44 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cdc19b94-fd21-4f32-84fe-b3afc5905b7b_calibrate_1124707_20.stdout parsl.dataflow.dflow INFO: Standard error for task 44 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cdc19b94-fd21-4f32-84fe-b3afc5905b7b_calibrate_1124707_20.stderr parsl.dataflow.dflow INFO: Task 863 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 691031f50695038f236a02c12a2dc80f with result from task 863 parsl.dataflow.dflow DEBUG: Task 863 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/b093d3e4-fd60-4b76-944a-35b0fffb943e_calibrate_1124707_28.stdout parsl.dataflow.dflow INFO: Standard error for task 863 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/b093d3e4-fd60-4b76-944a-35b0fffb943e_calibrate_1124707_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:25 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 32 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d55c274452fc726213cedb01ae3052a4 with result from task 32 parsl.dataflow.dflow DEBUG: Task 32 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/2b879117-db26-4824-9c43-2ee3a76a7559_calibrate_1124710_16.stdout parsl.dataflow.dflow INFO: Standard error for task 32 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/2b879117-db26-4824-9c43-2ee3a76a7559_calibrate_1124710_16.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:25 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 89 parsl.dataflow.strategy DEBUG: Executor multi has 89 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 3: no changes necessary to current block load parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:30:26 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:30:26 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 293 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ea06e7626690c34767c8a756a0a46824 with result from task 293 parsl.dataflow.dflow DEBUG: Task 293 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e50bd4df-9ec3-4921-8e00-e35e543f7089_calibrate_1124707_39.stdout parsl.dataflow.dflow INFO: Standard error for task 293 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e50bd4df-9ec3-4921-8e00-e35e543f7089_calibrate_1124707_39.stderr parsl.dataflow.dflow INFO: Task 689 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d6f9e2461620e8152fd09880d5c7f22 with result from task 689 parsl.dataflow.dflow DEBUG: Task 689 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ae522dc7-a361-46e7-b839-972d6a5f59c9_calibrate_1124704_21.stdout parsl.dataflow.dflow INFO: Standard error for task 689 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/ae522dc7-a361-46e7-b839-972d6a5f59c9_calibrate_1124704_21.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:26 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:26 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 308 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 923e93ae896bee510feabeb2cbd94179 with result from task 308 parsl.dataflow.dflow DEBUG: Task 308 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e0bf2b30-f9e1-4767-9d58-1459af999672_calibrate_1124707_60.stdout parsl.dataflow.dflow INFO: Standard error for task 308 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e0bf2b30-f9e1-4767-9d58-1459af999672_calibrate_1124707_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:27 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 839 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b4bbbbc4f1c0465297e6278770f05bad with result from task 839 parsl.dataflow.dflow DEBUG: Task 839 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/045bb14c-af03-44a5-8888-c618d2cd0d7c_calibrate_1124710_14.stdout parsl.dataflow.dflow INFO: Standard error for task 839 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/045bb14c-af03-44a5-8888-c618d2cd0d7c_calibrate_1124710_14.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:27 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 332 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 94fae6f175c508f8a9acd5d06850b013 with result from task 332 parsl.dataflow.dflow DEBUG: Task 332 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/b032e008-4347-42e9-8477-d752c6a7e20b_calibrate_1124704_12.stdout parsl.dataflow.dflow INFO: Standard error for task 332 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/b032e008-4347-42e9-8477-d752c6a7e20b_calibrate_1124704_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:27 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 767 try 1 failed parsl.dataflow.dflow ERROR: Task 767 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry c715fbad0dfe0df5530a38b18d4a0e92 with result from task 767 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/46d083a0-25ac-438f-ac7e-fa79f51f9dd8_calibrate_1124704_31.stdout parsl.dataflow.dflow INFO: Standard error for task 767 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/46d083a0-25ac-438f-ac7e-fa79f51f9dd8_calibrate_1124704_31.stderr parsl.dataflow.dflow INFO: Task 719 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c9a4c025a51a358ed10767b2e0eaed5 with result from task 719 parsl.dataflow.dflow DEBUG: Task 719 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/edfff04a-74b3-49c5-9a5f-1d8f048da000_calibrate_1124707_13.stdout parsl.dataflow.dflow INFO: Standard error for task 719 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/edfff04a-74b3-49c5-9a5f-1d8f048da000_calibrate_1124707_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:28 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 167 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7bac7785d0c32a44c5424d373ddd0930 with result from task 167 parsl.dataflow.dflow DEBUG: Task 167 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/72a47e40-27e5-436c-8c84-99e14e8c5b38_calibrate_1124704_6.stdout parsl.dataflow.dflow INFO: Standard error for task 167 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/72a47e40-27e5-436c-8c84-99e14e8c5b38_calibrate_1124704_6.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:28 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:28 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 86 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0505f7930b14683c69a0220b00990e51 with result from task 86 parsl.dataflow.dflow DEBUG: Task 86 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7000406f-7cc7-4ca8-866c-560e312af954_calibrate_1124710_55.stdout parsl.dataflow.dflow INFO: Standard error for task 86 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7000406f-7cc7-4ca8-866c-560e312af954_calibrate_1124710_55.stderr parsl.dataflow.dflow INFO: Task 821 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 55bf9172715b9c0eb0a8c420115edf56 with result from task 821 parsl.dataflow.dflow DEBUG: Task 821 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/fa205240-df9c-4c01-9105-d40fa1521ce7_calibrate_1124707_52.stdout parsl.dataflow.dflow INFO: Standard error for task 821 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/fa205240-df9c-4c01-9105-d40fa1521ce7_calibrate_1124707_52.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:28 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 701 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 37dc7f1b17045bd9031815ebd160fe4f with result from task 701 parsl.dataflow.dflow DEBUG: Task 701 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/aea468ce-e71e-4fd3-ae88-53be4400f6a3_calibrate_1124704_16.stdout parsl.dataflow.dflow INFO: Standard error for task 701 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/aea468ce-e71e-4fd3-ae88-53be4400f6a3_calibrate_1124704_16.stderr parsl.dataflow.dflow INFO: Task 290 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 479715092e37b2986eb405639cfc3b7b with result from task 290 parsl.dataflow.dflow DEBUG: Task 290 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/0f87c0c3-ae36-41c9-8937-9d46fd4badb9_calibrate_1124710_28.stdout parsl.dataflow.dflow INFO: Standard error for task 290 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/0f87c0c3-ae36-41c9-8937-9d46fd4badb9_calibrate_1124710_28.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:29 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:29 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 461 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 50e1e8ccb74eb426f0478d17e95b4d18 with result from task 461 parsl.dataflow.dflow DEBUG: Task 461 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/983f364a-1011-4e6e-91e1-67fd8f70e2be_calibrate_1124710_52.stdout parsl.dataflow.dflow INFO: Standard error for task 461 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/983f364a-1011-4e6e-91e1-67fd8f70e2be_calibrate_1124710_52.stderr parsl.dataflow.dflow INFO: Task 35 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 295d8e24c58446b4cb86e115c6933678 with result from task 35 parsl.dataflow.dflow DEBUG: Task 35 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/080030d8-7ae4-45a9-a4bc-8e007a14aa64_calibrate_1124710_25.stdout parsl.dataflow.dflow INFO: Standard error for task 35 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/080030d8-7ae4-45a9-a4bc-8e007a14aa64_calibrate_1124710_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:29 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:30 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:30 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:30 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:31 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 743 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8fe38dedcf049ddad4a3a2cc7f9a8bc6 with result from task 743 parsl.dataflow.dflow DEBUG: Task 743 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/3d3bf46f-df3f-4ecb-a663-cbb89782424a_calibrate_1124704_24.stdout parsl.dataflow.dflow INFO: Standard error for task 743 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/3d3bf46f-df3f-4ecb-a663-cbb89782424a_calibrate_1124704_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:31 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 341 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d5994f0617d45d5b0635f28f3be96652 with result from task 341 parsl.dataflow.dflow DEBUG: Task 341 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ef9c1552-afaa-4c22-8b3f-45ae2fd52883_calibrate_1124707_34.stdout parsl.dataflow.dflow INFO: Standard error for task 341 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/ef9c1552-afaa-4c22-8b3f-45ae2fd52883_calibrate_1124707_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:31 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 20 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc8ef5a44a42251ca0e13b07e76bbaff with result from task 20 parsl.dataflow.dflow DEBUG: Task 20 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/465da77e-a853-4003-8a44-8027012c10fc_calibrate_1124710_37.stdout parsl.dataflow.dflow INFO: Standard error for task 20 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/465da77e-a853-4003-8a44-8027012c10fc_calibrate_1124710_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:31 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 191 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8b832784b3a78c952c2ce11ea220f855 with result from task 191 parsl.dataflow.dflow DEBUG: Task 191 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/64017345-fc07-4b33-9e6c-e7959f03b047_calibrate_1124707_27.stdout parsl.dataflow.dflow INFO: Standard error for task 191 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/64017345-fc07-4b33-9e6c-e7959f03b047_calibrate_1124707_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:32 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:32 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 395 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e2b2ddc3c4fee455d986bbd3586a07a8 with result from task 395 parsl.dataflow.dflow DEBUG: Task 395 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ef59c1e1-6726-41f8-9d87-42fa659ebcbc_calibrate_1124710_48.stdout parsl.dataflow.dflow INFO: Standard error for task 395 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ef59c1e1-6726-41f8-9d87-42fa659ebcbc_calibrate_1124710_48.stderr parsl.dataflow.dflow INFO: Task 914 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry db04dcf7e14d5f1bf067a941127d92ea with result from task 914 parsl.dataflow.dflow DEBUG: Task 914 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ef82e583-965b-4e8d-a7bd-716e780e836b_calibrate_1124710_57.stdout parsl.dataflow.dflow INFO: Standard error for task 914 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ef82e583-965b-4e8d-a7bd-716e780e836b_calibrate_1124710_57.stderr parsl.dataflow.dflow INFO: Task 236 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 33dc930996343adde85bb4ca6515b3d5 with result from task 236 parsl.dataflow.dflow DEBUG: Task 236 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8a1a315a-4e76-4309-9090-a49e2a13161e_calibrate_1124704_43.stdout parsl.dataflow.dflow INFO: Standard error for task 236 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8a1a315a-4e76-4309-9090-a49e2a13161e_calibrate_1124704_43.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:32 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:33 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 449 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 595b3337feab9bf4ebbf7d935cc996de with result from task 449 parsl.dataflow.dflow DEBUG: Task 449 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/9734fcd8-259c-42a0-8361-744c66ace823_calibrate_1124707_54.stdout parsl.dataflow.dflow INFO: Standard error for task 449 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/9734fcd8-259c-42a0-8361-744c66ace823_calibrate_1124707_54.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:33 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 410 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cc9406f90c86ef6eae25651434f6546c with result from task 410 parsl.dataflow.dflow DEBUG: Task 410 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/35c94c10-32b6-4796-907c-8bfc0119f8f0_calibrate_1124707_48.stdout parsl.dataflow.dflow INFO: Standard error for task 410 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/35c94c10-32b6-4796-907c-8bfc0119f8f0_calibrate_1124707_48.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:33 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:34 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 575 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 96a420cbf075152ae21b823ef0cf3612 with result from task 575 parsl.dataflow.dflow DEBUG: Task 575 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/7ee0e280-24cc-4759-954f-f23974918725_calibrate_1124704_42.stdout parsl.dataflow.dflow INFO: Standard error for task 575 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/7ee0e280-24cc-4759-954f-f23974918725_calibrate_1124704_42.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:34 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 755 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d80f3fa96dd42027bf8af5ddda15874b with result from task 755 parsl.dataflow.dflow DEBUG: Task 755 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/86c4e160-796c-4069-8071-fd2e86cccf19_calibrate_1124704_38.stdout parsl.dataflow.dflow INFO: Standard error for task 755 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/86c4e160-796c-4069-8071-fd2e86cccf19_calibrate_1124704_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:34 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:34 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 485 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fc96c72b7eb2d17d9a0e3b9268169fb0 with result from task 485 parsl.dataflow.dflow DEBUG: Task 485 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/95b85a38-ed3a-4871-8ba9-bac964c14f8b_calibrate_1124710_34.stdout parsl.dataflow.dflow INFO: Standard error for task 485 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/95b85a38-ed3a-4871-8ba9-bac964c14f8b_calibrate_1124710_34.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:35 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 389 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry bd87eb0d204fa4b9f0179b02df9ae346 with result from task 389 parsl.dataflow.dflow DEBUG: Task 389 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/babc7555-1937-46b1-8b76-20bfbb1417d3_calibrate_1124704_7.stdout parsl.dataflow.dflow INFO: Standard error for task 389 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/babc7555-1937-46b1-8b76-20bfbb1417d3_calibrate_1124704_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:35 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 905 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry fb75ec92a74f9c34bedd0510023b88e4 with result from task 905 parsl.dataflow.dflow DEBUG: Task 905 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/521c2687-a0ad-43a3-9c19-0ad8ef9f522b_calibrate_1124707_1.stdout parsl.dataflow.dflow INFO: Standard error for task 905 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/521c2687-a0ad-43a3-9c19-0ad8ef9f522b_calibrate_1124707_1.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:35 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:36 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.dataflow.dflow INFO: Task 902 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5dc8d02be02b5127cd85bae5e3bb279a with result from task 902 parsl.dataflow.dflow DEBUG: Task 902 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/fdec7c70-6bf5-4f35-9ac0-567c652f5bcb_calibrate_1124704_23.stdout parsl.dataflow.dflow INFO: Standard error for task 902 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/fdec7c70-6bf5-4f35-9ac0-567c652f5bcb_calibrate_1124704_23.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:36 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:36 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:36 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:37 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:37 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:37 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:37 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 710 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 673928048421be2f4a46d33acf44e959 with result from task 710 parsl.dataflow.dflow DEBUG: Task 710 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/84967446-06d1-4edd-9d06-c1714e629f0b_calibrate_1124707_33.stdout parsl.dataflow.dflow INFO: Standard error for task 710 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/84967446-06d1-4edd-9d06-c1714e629f0b_calibrate_1124707_33.stderr parsl.dataflow.dflow INFO: Task 635 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 956aa95406b5eef9473c9b8af096eca5 with result from task 635 parsl.dataflow.dflow DEBUG: Task 635 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/bff99b4a-25f1-4b14-80a9-411e53b57b8e_calibrate_1124710_21.stdout parsl.dataflow.dflow INFO: Standard error for task 635 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/bff99b4a-25f1-4b14-80a9-411e53b57b8e_calibrate_1124710_21.stderr parsl.dataflow.dflow INFO: Task 38 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4f5b8372c818f4ac9476aa52f7b98485 with result from task 38 parsl.dataflow.dflow DEBUG: Task 38 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/2ea5cb51-2149-49c6-8189-52fcbaa11c74_calibrate_1124707_25.stdout parsl.dataflow.dflow INFO: Standard error for task 38 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/2ea5cb51-2149-49c6-8189-52fcbaa11c74_calibrate_1124707_25.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:38 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 254 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry d9f8a54176a625e80d058273c5b5eab1 with result from task 254 parsl.dataflow.dflow DEBUG: Task 254 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/75fe4f29-b18b-4b69-ace4-f51a7b7b9a92_calibrate_1124704_18.stdout parsl.dataflow.dflow INFO: Standard error for task 254 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/75fe4f29-b18b-4b69-ace4-f51a7b7b9a92_calibrate_1124704_18.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:38 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:38 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 227 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 36e5337edac2418694c71a3abaf8f0bc with result from task 227 parsl.dataflow.dflow DEBUG: Task 227 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/fbae5cf0-c419-49c2-92a1-022648a57d90_calibrate_1124707_37.stdout parsl.dataflow.dflow INFO: Standard error for task 227 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/fbae5cf0-c419-49c2-92a1-022648a57d90_calibrate_1124707_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:38 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:39 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 761 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry da2acf08c3978089827c89def71446f2 with result from task 761 parsl.dataflow.dflow DEBUG: Task 761 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8800fe4f-1870-401a-a66c-6d4f67f9f11b_calibrate_1124704_51.stdout parsl.dataflow.dflow INFO: Standard error for task 761 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8800fe4f-1870-401a-a66c-6d4f67f9f11b_calibrate_1124704_51.stderr parsl.dataflow.dflow INFO: Task 674 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a85c9c54af0f3ae1b9150f5ff06d9062 with result from task 674 parsl.dataflow.dflow DEBUG: Task 674 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/63c7a0f9-d8dc-4a7d-887d-ac2fd37fd1e1_calibrate_1124704_55.stdout parsl.dataflow.dflow INFO: Standard error for task 674 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/63c7a0f9-d8dc-4a7d-887d-ac2fd37fd1e1_calibrate_1124704_55.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:39 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:39 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 248 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry ae13f9c21ed608b1311659755659794d with result from task 248 parsl.dataflow.dflow DEBUG: Task 248 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/2e1f458a-47af-48f9-9abb-3b1f89fc0b7f_calibrate_1124704_40.stdout parsl.dataflow.dflow INFO: Standard error for task 248 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/2e1f458a-47af-48f9-9abb-3b1f89fc0b7f_calibrate_1124704_40.stderr parsl.dataflow.dflow INFO: Task 797 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4d3c800a346dcfcc4f2d0d000980c086 with result from task 797 parsl.dataflow.dflow DEBUG: Task 797 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/927f6d5e-5aad-48c8-bc6a-16e772cc1e22_calibrate_1124710_13.stdout parsl.dataflow.dflow INFO: Standard error for task 797 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/927f6d5e-5aad-48c8-bc6a-16e772cc1e22_calibrate_1124710_13.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:39 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:40 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:40 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 158 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 8a4a422a9393f81fbfdd4c1b83645d17 with result from task 158 parsl.dataflow.dflow DEBUG: Task 158 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/cdd0acc1-1054-437b-94a1-c6ffad16d57d_calibrate_1124710_56.stdout parsl.dataflow.dflow INFO: Standard error for task 158 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/cdd0acc1-1054-437b-94a1-c6ffad16d57d_calibrate_1124710_56.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:40 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 794 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c7ee6db9b3b197f677de0f8f94295c29 with result from task 794 parsl.dataflow.dflow DEBUG: Task 794 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/3130fa52-7612-45de-98ec-2b296ee56466_calibrate_1124707_45.stdout parsl.dataflow.dflow INFO: Standard error for task 794 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/3130fa52-7612-45de-98ec-2b296ee56466_calibrate_1124707_45.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:40 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 898 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 952f1537eabc0300c3b4c0ba0c84c6e0 with result from task 898 parsl.dataflow.dflow DEBUG: Task 898 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 899 has memoization hash 667d182c95a4576c8737f8ad7ca8c2cd parsl.dataflow.memoization INFO: Task 899 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352960bb00> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 899 try 0 launched on executor multi with executor id 927 parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/e01a31e7-e6f9-4860-8ad3-8483e774e31b_calibrate_1124704_44.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/e01a31e7-e6f9-4860-8ad3-8483e774e31b_calibrate_1124704_44.stderr parsl.dataflow.dflow INFO: Standard output for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/4108c9f4-94af-4b52-8374-e16303d63664_characterizeImage_1124704_44.stdout parsl.dataflow.dflow INFO: Standard error for task 898 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/4108c9f4-94af-4b52-8374-e16303d63664_characterizeImage_1124704_44.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:40 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:41 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:41 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 815 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 558c898a14b0e1214b132fef4cb80954 with result from task 815 parsl.dataflow.dflow DEBUG: Task 815 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1be2fec9-b591-41a2-900d-926b370f2507_calibrate_1124710_38.stdout parsl.dataflow.dflow INFO: Standard error for task 815 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/1be2fec9-b591-41a2-900d-926b370f2507_calibrate_1124710_38.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:41 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:41 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:42 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 488 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 78f889c3f0d8f61fcdfa790bb7f0021e with result from task 488 parsl.dataflow.dflow DEBUG: Task 488 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/6579d12f-21e2-4f18-b564-f123bbf035cc_calibrate_1124707_51.stdout parsl.dataflow.dflow INFO: Standard error for task 488 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/6579d12f-21e2-4f18-b564-f123bbf035cc_calibrate_1124707_51.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:42 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 473 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 370e63755df3047ab63629ebc81109da with result from task 473 parsl.dataflow.dflow DEBUG: Task 473 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/5f12ade6-fdff-4c93-808b-d7c41cc9a482_calibrate_1124710_47.stdout parsl.dataflow.dflow INFO: Standard error for task 473 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/5f12ade6-fdff-4c93-808b-d7c41cc9a482_calibrate_1124710_47.stderr parsl.dataflow.dflow INFO: Task 827 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0408185fd595f65ab2f6dee326701fac with result from task 827 parsl.dataflow.dflow DEBUG: Task 827 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/de8d0c87-543a-4416-a655-76e5f4208abb_calibrate_1124704_32.stdout parsl.dataflow.dflow INFO: Standard error for task 827 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/de8d0c87-543a-4416-a655-76e5f4208abb_calibrate_1124704_32.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:42 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:42 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:43 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 455 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 91d0135722e35aba63b9445c674019bc with result from task 455 parsl.dataflow.dflow DEBUG: Task 455 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/229dcf9e-d025-4928-a897-fac59ff51cb7_calibrate_1124710_12.stdout parsl.dataflow.dflow INFO: Standard error for task 455 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/229dcf9e-d025-4928-a897-fac59ff51cb7_calibrate_1124710_12.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:43 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 881 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9b631e9a3cce320a821f2aa49bc7c00c with result from task 881 parsl.dataflow.dflow DEBUG: Task 881 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/70ac579c-d8d3-4430-a213-a37cf4ea14fb_calibrate_1124704_54.stdout parsl.dataflow.dflow INFO: Standard error for task 881 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/70ac579c-d8d3-4430-a213-a37cf4ea14fb_calibrate_1124704_54.stderr parsl.dataflow.dflow INFO: Task 509 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 66cc1d675ec5f00c3bb6e6b902931c29 with result from task 509 parsl.dataflow.dflow DEBUG: Task 509 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/5ceab459-6ca7-48fc-986f-fb9684d491e3_calibrate_1124704_3.stdout parsl.dataflow.dflow INFO: Standard error for task 509 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/5ceab459-6ca7-48fc-986f-fb9684d491e3_calibrate_1124704_3.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:43 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:43 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:44 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:44 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:44 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:44 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:45 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:45 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:45 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:45 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 284 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 3c79d2b25ae0efe7ebfac3514c3f77aa with result from task 284 parsl.dataflow.dflow DEBUG: Task 284 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/20a869a7-d8fb-4770-ab0e-f4720fe60e68_calibrate_1124707_22.stdout parsl.dataflow.dflow INFO: Standard error for task 284 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/20a869a7-d8fb-4770-ab0e-f4720fe60e68_calibrate_1124707_22.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:45 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 245 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6ca111a9f953d7cd6ed4f9b6209d98a1 with result from task 245 parsl.dataflow.dflow DEBUG: Task 245 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/5c44a510-a23e-479e-a3b0-d327b0af5722_calibrate_1124704_27.stdout parsl.dataflow.dflow INFO: Standard error for task 245 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/5c44a510-a23e-479e-a3b0-d327b0af5722_calibrate_1124704_27.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:46 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:46 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 40 parsl.dataflow.strategy DEBUG: Executor multi has 40 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 40 parsl.dataflow.strategy DEBUG: Executor multi has 40 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.dflow INFO: Task 251 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 92d548251728ada8a09f10363f50dd40 with result from task 251 parsl.dataflow.dflow DEBUG: Task 251 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/e779a88d-9134-4799-99d0-12731bd5e0c2_calibrate_1124710_44.stdout parsl.dataflow.dflow INFO: Standard error for task 251 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/e779a88d-9134-4799-99d0-12731bd5e0c2_calibrate_1124710_44.stderr parsl.dataflow.dflow INFO: Task 209 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 179ff4acff7366e74408592d697aac43 with result from task 209 parsl.dataflow.dflow DEBUG: Task 209 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/cf783236-09f4-4afb-af80-b673ab1a6b41_calibrate_1124710_46.stdout parsl.dataflow.dflow INFO: Standard error for task 209 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/cf783236-09f4-4afb-af80-b673ab1a6b41_calibrate_1124710_46.stderr parsl.dataflow.dflow INFO: Task 503 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be2a83c8d42a87e1f6cc10c6b3510e99 with result from task 503 parsl.dataflow.dflow DEBUG: Task 503 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/0769f45c-e3b0-422a-aee7-561788fd79fb_calibrate_1124707_53.stdout parsl.dataflow.dflow INFO: Standard error for task 503 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/0769f45c-e3b0-422a-aee7-561788fd79fb_calibrate_1124707_53.stderr parsl.dataflow.dflow INFO: Task 662 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c85cf9ac71c7b3b1530c9246a4d7b840 with result from task 662 parsl.dataflow.dflow DEBUG: Task 662 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/926b33b3-5a43-4374-9ee2-791a03c29e24_calibrate_1124710_49.stdout parsl.dataflow.dflow INFO: Standard error for task 662 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/926b33b3-5a43-4374-9ee2-791a03c29e24_calibrate_1124710_49.stderr parsl.dataflow.dflow INFO: Task 848 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry dbb81b14303e9b5ee6ae25b4fae0d699 with result from task 848 parsl.dataflow.dflow DEBUG: Task 848 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/feb963bf-87e5-4cfc-a9a1-1c935a042ff9_calibrate_1124707_59.stdout parsl.dataflow.dflow INFO: Standard error for task 848 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/feb963bf-87e5-4cfc-a9a1-1c935a042ff9_calibrate_1124707_59.stderr parsl.dataflow.dflow INFO: Task 479 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry cd9df7a66c64921eb5fc0618f5113894 with result from task 479 parsl.dataflow.dflow DEBUG: Task 479 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/9fcb1502-5980-48df-a306-410a38b6e972_calibrate_1124704_11.stdout parsl.dataflow.dflow INFO: Standard error for task 479 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/9fcb1502-5980-48df-a306-410a38b6e972_calibrate_1124704_11.stderr parsl.dataflow.dflow INFO: Task 893 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b16e8fa2fd397240e0cdc26fe9bb4e06 with result from task 893 parsl.dataflow.dflow DEBUG: Task 893 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/27c4ec59-233e-4596-93b5-bfc72dce093a_calibrate_1124707_23.stdout parsl.dataflow.dflow INFO: Standard error for task 893 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/27c4ec59-233e-4596-93b5-bfc72dce093a_calibrate_1124707_23.stderr parsl.dataflow.dflow INFO: Task 653 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7c5bf1bb6a5a1782b35fdf387ea1d207 with result from task 653 parsl.dataflow.dflow DEBUG: Task 653 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e781d8d5-84d5-4ce2-8c65-048d42c20d01_calibrate_1124707_55.stdout parsl.dataflow.dflow INFO: Standard error for task 653 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e781d8d5-84d5-4ce2-8c65-048d42c20d01_calibrate_1124707_55.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 32 parsl.dataflow.strategy DEBUG: Executor multi has 32 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.dflow INFO: Task 497 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c974fec96921108be1860e9f2e53e529 with result from task 497 parsl.dataflow.dflow DEBUG: Task 497 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e267f1fb-7711-45ef-995d-ea5876523e83_calibrate_1124707_40.stdout parsl.dataflow.dflow INFO: Standard error for task 497 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/e267f1fb-7711-45ef-995d-ea5876523e83_calibrate_1124707_40.stderr parsl.dataflow.dflow INFO: Task 776 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry be6ca0e51228ed8777d72b18b6070677 with result from task 776 parsl.dataflow.dflow DEBUG: Task 776 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ff4c9252-51e9-4975-8666-2199fbb6ce33_calibrate_1124710_8.stdout parsl.dataflow.dflow INFO: Standard error for task 776 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/ff4c9252-51e9-4975-8666-2199fbb6ce33_calibrate_1124710_8.stderr parsl.dataflow.dflow INFO: Task 179 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7f4b8651b13755a60530eb60e3d38bbc with result from task 179 parsl.dataflow.dflow DEBUG: Task 179 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/26a723c0-e463-4053-a0c5-b28e5091fcbb_calibrate_1124704_60.stdout parsl.dataflow.dflow INFO: Standard error for task 179 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/26a723c0-e463-4053-a0c5-b28e5091fcbb_calibrate_1124704_60.stderr parsl.dataflow.dflow INFO: Task 359 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5774cb67f4f2d0dfbd687805402f0ddb with result from task 359 parsl.dataflow.dflow DEBUG: Task 359 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/290bb44a-f309-4176-98eb-018c6cd98529_calibrate_1124710_43.stdout parsl.dataflow.dflow INFO: Standard error for task 359 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/290bb44a-f309-4176-98eb-018c6cd98529_calibrate_1124710_43.stderr parsl.dataflow.dflow INFO: Task 872 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 684a572f5d0ab893a5971e3d04f3d8b5 with result from task 872 parsl.dataflow.dflow DEBUG: Task 872 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/f1380197-c011-4826-978b-1d8b3b4b8b8a_calibrate_1124704_15.stdout parsl.dataflow.dflow INFO: Standard error for task 872 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/f1380197-c011-4826-978b-1d8b3b4b8b8a_calibrate_1124704_15.stderr parsl.dataflow.dflow INFO: Task 770 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e87cf48b3eb63279c6c3f87b84f0ad71 with result from task 770 parsl.dataflow.dflow DEBUG: Task 770 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/2efc6732-25ab-49f8-9759-ec06e1886f24_calibrate_1124704_59.stdout parsl.dataflow.dflow INFO: Standard error for task 770 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/2efc6732-25ab-49f8-9759-ec06e1886f24_calibrate_1124704_59.stderr parsl.dataflow.dflow INFO: Task 29 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 0d0fb87c4d5ef8426ad43aedb68b9a28 with result from task 29 parsl.dataflow.dflow DEBUG: Task 29 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/5614919d-83e8-4c19-8780-0e0bfe4c026b_calibrate_1124707_49.stdout parsl.dataflow.dflow INFO: Standard error for task 29 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/5614919d-83e8-4c19-8780-0e0bfe4c026b_calibrate_1124707_49.stderr parsl.dataflow.dflow INFO: Task 173 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e421a711bcb804f8b9da49f6e74512de with result from task 173 parsl.dataflow.dflow DEBUG: Task 173 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/609c82be-6502-4b6b-a303-f9425d098b68_calibrate_1124707_30.stdout parsl.dataflow.dflow INFO: Standard error for task 173 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/609c82be-6502-4b6b-a303-f9425d098b68_calibrate_1124707_30.stderr parsl.dataflow.dflow INFO: Task 908 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 32e1b8e6e62be72084363c885878face with result from task 908 parsl.dataflow.dflow DEBUG: Task 908 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/59f61e14-e803-4cee-80ca-420c66a7160b_calibrate_1124707_50.stdout parsl.dataflow.dflow INFO: Standard error for task 908 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/59f61e14-e803-4cee-80ca-420c66a7160b_calibrate_1124707_50.stderr parsl.dataflow.dflow INFO: Task 182 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 2cecae0e9f2e62e3049f855d21ec845a with result from task 182 parsl.dataflow.dflow DEBUG: Task 182 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/51e2f4cf-4670-4bd2-b532-5eb4b28f8529_calibrate_1124704_58.stdout parsl.dataflow.dflow INFO: Standard error for task 182 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/51e2f4cf-4670-4bd2-b532-5eb4b28f8529_calibrate_1124704_58.stderr parsl.dataflow.dflow INFO: Task 206 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 22e0072dbee3eef026c12c865902521e with result from task 206 parsl.dataflow.dflow DEBUG: Task 206 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/48caa663-0ac4-4e9b-9edd-83ebff6c7133_calibrate_1124704_28.stdout parsl.dataflow.dflow INFO: Standard error for task 206 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/48caa663-0ac4-4e9b-9edd-83ebff6c7133_calibrate_1124704_28.stderr parsl.dataflow.dflow INFO: Task 722 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 181e5c29cec5617400d34bf5694be04f with result from task 722 parsl.dataflow.dflow DEBUG: Task 722 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/65923946-7a79-4bf5-b21a-f7f6508cbd8a_calibrate_1124707_44.stdout parsl.dataflow.dflow INFO: Standard error for task 722 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/65923946-7a79-4bf5-b21a-f7f6508cbd8a_calibrate_1124707_44.stderr parsl.dataflow.dflow INFO: Task 890 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8984f3b8ebee3e8e62db4b47f649831 with result from task 890 parsl.dataflow.dflow DEBUG: Task 890 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/55c272be-89bf-4b21-bdbc-74a3f11deb26_calibrate_1124707_29.stdout parsl.dataflow.dflow INFO: Standard error for task 890 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/55c272be-89bf-4b21-bdbc-74a3f11deb26_calibrate_1124707_29.stderr parsl.dataflow.dflow INFO: Task 476 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 81e52af518e524724e5e860c6f630b4b with result from task 476 parsl.dataflow.dflow DEBUG: Task 476 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/8c30d298-aa18-4619-8089-f2bb3d31aebc_calibrate_1124707_11.stdout parsl.dataflow.dflow INFO: Standard error for task 476 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/8c30d298-aa18-4619-8089-f2bb3d31aebc_calibrate_1124707_11.stderr 2024-03-07 07:30:56 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:30:56 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:56 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:57 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:57 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:57 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:57 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:58 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:58 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:58 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:58 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:58 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:59 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:59 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:59 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 548 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 6e0264845d5d0abe388b9aa9f638304b with result from task 548 parsl.dataflow.dflow DEBUG: Task 548 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/12f479a3-9f88-4190-b4b2-25b0f77d967d_calibrate_1124710_4.stdout parsl.dataflow.dflow INFO: Standard error for task 548 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/12f479a3-9f88-4190-b4b2-25b0f77d967d_calibrate_1124710_4.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:59 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:30:59 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:00 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:00 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 713 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry a8a205eeee6d170ed07c465f2aa08aff with result from task 713 parsl.dataflow.dflow DEBUG: Task 713 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/c657083d-c8cc-4671-9619-3bd5f4283cda_calibrate_1124710_40.stdout parsl.dataflow.dflow INFO: Standard error for task 713 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/c657083d-c8cc-4671-9619-3bd5f4283cda_calibrate_1124710_40.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:00 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:00 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:00 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:01 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 164 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 5a415a01f53634653abd16e273be9f7a with result from task 164 parsl.dataflow.dflow DEBUG: Task 164 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/12af302b-c2eb-4c8e-b809-e3a1b1b7ba3d_calibrate_1124704_37.stdout parsl.dataflow.dflow INFO: Standard error for task 164 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/12af302b-c2eb-4c8e-b809-e3a1b1b7ba3d_calibrate_1124704_37.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:01 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:01 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:01 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:01 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:02 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:02 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 452 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 41fbce7696e8b7f5f58d3f233ed9071c with result from task 452 parsl.dataflow.dflow DEBUG: Task 452 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/4ae27cd1-4a76-4a69-9e78-46bd15b4b07f_calibrate_1124710_60.stdout parsl.dataflow.dflow INFO: Standard error for task 452 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/4ae27cd1-4a76-4a69-9e78-46bd15b4b07f_calibrate_1124710_60.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:02 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:02 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:02 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:03 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:03 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:03 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:03 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:03 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:04 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:04 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:04 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:04 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:04 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:05 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:05 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:05 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:05 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:05 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:06 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:06 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:06 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:06 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:07 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:07 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:07 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:07 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:07 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:08 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:08 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:08 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:08 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:08 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:09 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:09 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:09 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:09 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:09 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:10 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:10 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:10 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:10 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 278 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 1e30062eeb161e72a65902ab846b05d0 with result from task 278 parsl.dataflow.dflow DEBUG: Task 278 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1ba10e4c-6a44-44d9-9aa9-4a2678062525_calibrate_1124707_24.stdout parsl.dataflow.dflow INFO: Standard error for task 278 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/1ba10e4c-6a44-44d9-9aa9-4a2678062525_calibrate_1124707_24.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:10 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:11 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:11 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:11 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:11 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:11 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:12 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:12 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.dataflow.dflow INFO: Task 203 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 52a1d1a680b139a5670d07d0e028e905 with result from task 203 parsl.dataflow.dflow DEBUG: Task 203 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/bed14d87-2dc3-4e58-aed3-1c8c799dbdea_calibrate_1124707_17.stdout parsl.dataflow.dflow INFO: Standard error for task 203 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/bed14d87-2dc3-4e58-aed3-1c8c799dbdea_calibrate_1124707_17.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 12 parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 12 parsl.dataflow.strategy DEBUG: Executor multi has 12 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.dflow INFO: Task 170 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry f7fe7cd42245389c46994386eb26bb18 with result from task 170 parsl.dataflow.dflow DEBUG: Task 170 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8b83b24a-0c75-48be-b1e1-eb0d459d3853_calibrate_1124704_25.stdout parsl.dataflow.dflow INFO: Standard error for task 170 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/8b83b24a-0c75-48be-b1e1-eb0d459d3853_calibrate_1124704_25.stderr parsl.dataflow.dflow INFO: Task 800 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c0045da3f8752bde0a1ec0b898794d4e with result from task 800 parsl.dataflow.dflow DEBUG: Task 800 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7ab9af8c-7f53-466f-94ff-a5243e5b190d_calibrate_1124710_7.stdout parsl.dataflow.dflow INFO: Standard error for task 800 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/7ab9af8c-7f53-466f-94ff-a5243e5b190d_calibrate_1124710_7.stderr parsl.dataflow.dflow INFO: Task 896 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4feb3ca7dab65154040d1fbdfee23ac1 with result from task 896 parsl.dataflow.dflow DEBUG: Task 896 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/20927db7-5ea3-49cc-8f82-fd6209a2a39a_calibrate_1124710_59.stdout parsl.dataflow.dflow INFO: Standard error for task 896 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/20927db7-5ea3-49cc-8f82-fd6209a2a39a_calibrate_1124710_59.stderr parsl.dataflow.dflow INFO: Task 920 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e662d26ec432b5ba2301844b4f38d51e with result from task 920 parsl.dataflow.dflow DEBUG: Task 920 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/a82a9349-d637-4143-824c-78ed263b7c73_calibrate_1124704_48.stdout parsl.dataflow.dflow INFO: Standard error for task 920 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/a82a9349-d637-4143-824c-78ed263b7c73_calibrate_1124704_48.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.dflow INFO: Task 262 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e77436a85fe8af2356fdae9b1c0c3c14 with result from task 262 parsl.dataflow.dflow DEBUG: Task 262 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 263 has memoization hash e7fa934a38bffb7487f0b34e2a81ae4a parsl.dataflow.memoization INFO: Task 263 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297aa200> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 263 try 0 launched on executor multi with executor id 928 parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/9345d44c-650b-455e-bf12-7fa3186eb545_calibrate_1124707_10.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/9345d44c-650b-455e-bf12-7fa3186eb545_calibrate_1124707_10.stderr parsl.dataflow.dflow INFO: Standard output for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/32bac25f-97c6-4233-a005-57df9735bf0e_characterizeImage_1124707_10.stdout parsl.dataflow.dflow INFO: Standard error for task 262 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/32bac25f-97c6-4233-a005-57df9735bf0e_characterizeImage_1124707_10.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 8 parsl.dataflow.strategy DEBUG: Executor multi has 8 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.dflow INFO: Task 899 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 667d182c95a4576c8737f8ad7ca8c2cd with result from task 899 parsl.dataflow.dflow DEBUG: Task 899 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/e01a31e7-e6f9-4860-8ad3-8483e774e31b_calibrate_1124704_44.stdout parsl.dataflow.dflow INFO: Standard error for task 899 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/e01a31e7-e6f9-4860-8ad3-8483e774e31b_calibrate_1124704_44.stderr 2024-03-07 07:31:27 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:31:27 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:27 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:28 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 758 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry b58e169f260f94b7ae05363a9ef95543 with result from task 758 parsl.dataflow.dflow DEBUG: Task 758 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cbd6f5c1-f71c-4d37-93ee-d5a48bdd33c9_calibrate_1124707_7.stdout parsl.dataflow.dflow INFO: Standard error for task 758 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/cbd6f5c1-f71c-4d37-93ee-d5a48bdd33c9_calibrate_1124707_7.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:28 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:29 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:29 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:30 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:30 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:30 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:30 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:30 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:31 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:31 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:31 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:31 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:31 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:32 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:32 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:32 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:32 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:32 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:33 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:33 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:33 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:33 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:34 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:34 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:34 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:34 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:34 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:35 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:35 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:35 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:35 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:35 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:36 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:36 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:36 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:36 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:36 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:37 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:37 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:37 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:37 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:37 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:38 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:38 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:38 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:38 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:38 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:39 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:39 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:39 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:39 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:39 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:40 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:40 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:40 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:40 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:40 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:41 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:41 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:41 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:41 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:42 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:42 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:42 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:42 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:42 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:43 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:43 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:43 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:43 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:43 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:44 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:44 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:31:44 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 6 parsl.dataflow.strategy DEBUG: Executor multi has 6 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:31:59 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:31:59 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:01 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:01 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:02 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:03 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:03 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:04 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:04 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:04 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:04 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:04 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:05 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:05 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:05 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:05 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:05 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:06 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:06 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:06 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:06 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:07 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:07 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:07 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:07 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:07 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:08 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:08 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:08 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:08 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:08 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:09 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:09 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:09 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:09 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:09 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:10 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:10 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:10 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:10 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:10 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:11 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 263 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e7fa934a38bffb7487f0b34e2a81ae4a with result from task 263 parsl.dataflow.dflow DEBUG: Task 263 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/9345d44c-650b-455e-bf12-7fa3186eb545_calibrate_1124707_10.stdout parsl.dataflow.dflow INFO: Standard error for task 263 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/9345d44c-650b-455e-bf12-7fa3186eb545_calibrate_1124707_10.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:11 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:11 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:11 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:11 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:11 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:12 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:12 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:12 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:12 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:12 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:12 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:13 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:13 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:13 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:13 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:13 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:13 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:13 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:14 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:14 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:14 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:14 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:14 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:14 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:15 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:15 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:15 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:15 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:15 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:15 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:15 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:16 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:16 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:16 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:16 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:16 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.dflow INFO: Task 715 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry c8b825bc5a592437df6dfb13c873b017 with result from task 715 parsl.dataflow.dflow DEBUG: Task 715 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 5a8ee3ac0779c97a0da86ea9b9515c7b parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 0 launched on executor multi with executor id 929 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/d8e0940f-6b07-48c3-8ceb-00eb564cab37_calibrate_1124704_61.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/d8e0940f-6b07-48c3-8ceb-00eb564cab37_calibrate_1124704_61.stderr parsl.dataflow.dflow INFO: Standard output for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/e564d309-f127-416c-b404-961bef2a7de9_characterizeImage_1124704_61.stdout parsl.dataflow.dflow INFO: Standard error for task 715 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/e564d309-f127-416c-b404-961bef2a7de9_characterizeImage_1124704_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:32:31 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:32:31 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:32 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:32 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:32 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:32 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:32 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:32 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:33 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:33 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:33 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:33 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:33 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:35 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:35 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:36 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:36 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:37 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:37 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:37 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:38 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:38 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:38 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:38 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:38 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:38 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:39 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:39 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:39 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:39 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:39 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:39 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:39 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:40 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:40 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:40 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:40 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:40 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:40 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:40 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:41 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:41 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:41 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:41 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:41 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:41 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:42 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:42 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:42 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:42 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:42 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:42 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:42 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:43 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:43 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:43 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:43 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:43 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:43 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:44 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:44 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:44 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:44 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:44 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:44 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:44 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:45 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:45 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:45 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:45 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:45 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:45 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:46 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:46 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:46 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:46 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:46 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:32:46 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:33:01 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:33:02 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:02 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:02 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:02 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:02 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:02 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:02 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:03 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:03 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:03 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:03 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:03 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:03 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:04 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:04 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:04 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:04 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:04 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:04 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:04 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:05 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:05 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:05 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:05 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:05 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:05 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:06 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:06 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:06 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:06 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:06 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:06 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:06 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:07 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:07 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:07 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:07 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:09 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:09 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:10 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:10 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:10 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:10 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:10 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:10 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:10 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:11 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:11 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:11 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:11 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:11 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:11 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:12 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:12 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:12 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:12 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:12 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:12 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:12 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:13 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:13 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:13 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:13 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:13 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:13 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:14 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:14 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:14 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:14 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:14 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:14 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:14 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:15 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:15 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:15 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:15 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:15 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 716 try 0 failed parsl.dataflow.dflow INFO: Task 716 marked for retry parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/d8e0940f-6b07-48c3-8ceb-00eb564cab37_calibrate_1124704_61.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/d8e0940f-6b07-48c3-8ceb-00eb564cab37_calibrate_1124704_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 716 has memoization hash 5a8ee3ac0779c97a0da86ea9b9515c7b parsl.dataflow.memoization INFO: Task 716 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977c180> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 716 try 1 launched on executor multi with executor id 930 parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/d8e0940f-6b07-48c3-8ceb-00eb564cab37_calibrate_1124704_61.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/d8e0940f-6b07-48c3-8ceb-00eb564cab37_calibrate_1124704_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:33:35 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:33:36 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:36 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:36 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:36 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:36 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:36 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:36 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:37 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:37 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:37 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:37 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:37 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:37 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:38 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:38 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:38 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:38 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:38 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:38 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:39 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:39 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:39 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:39 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:39 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:39 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:39 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:40 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 784 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry acb35b91f995e2b25bf3316ea832b914 with result from task 784 parsl.dataflow.dflow DEBUG: Task 784 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 785 has memoization hash 7431daaa5b5d1a710e7dfd4beffc20ca parsl.dataflow.memoization INFO: Task 785 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535295c87c0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 785 try 0 launched on executor multi with executor id 931 parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/86526536-1e69-4d49-9ead-1659079add1d_calibrate_1124710_20.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/86526536-1e69-4d49-9ead-1659079add1d_calibrate_1124710_20.stderr parsl.dataflow.dflow INFO: Standard output for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7b26fa5d-acd5-449b-b1a5-1bfe3eed3e1c_characterizeImage_1124710_20.stdout parsl.dataflow.dflow INFO: Standard error for task 784 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/7b26fa5d-acd5-449b-b1a5-1bfe3eed3e1c_characterizeImage_1124710_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:42 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:42 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:42 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:44 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:44 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:44 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:44 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:44 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:45 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:45 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:45 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:45 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:46 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:46 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:46 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:46 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:47 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:47 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:47 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:47 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:47 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:48 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:48 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:48 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:48 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:48 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:49 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:49 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:49 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:49 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:49 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:50 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:50 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:50 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:50 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:50 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:51 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:51 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:51 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:51 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:51 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:52 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:52 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:52 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:52 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:53 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:53 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:53 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:53 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:33:53 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 5 parsl.dataflow.strategy DEBUG: Executor multi has 5 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:34:09 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:34:09 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:09 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:09 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:09 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:09 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:10 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:10 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:10 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:10 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:11 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:11 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:11 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:11 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:11 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:12 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:12 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:12 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:12 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:12 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:13 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:13 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:13 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:13 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:13 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:14 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:16 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:16 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:16 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:18 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:18 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:18 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:18 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:18 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:19 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:19 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:19 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:19 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:20 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:20 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:20 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:20 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:20 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 716 try 1 failed parsl.dataflow.dflow ERROR: Task 716 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5a8ee3ac0779c97a0da86ea9b9515c7b with result from task 716 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/d8e0940f-6b07-48c3-8ceb-00eb564cab37_calibrate_1124704_61.stdout parsl.dataflow.dflow INFO: Standard error for task 716 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/d8e0940f-6b07-48c3-8ceb-00eb564cab37_calibrate_1124704_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:20 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:21 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:21 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:21 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:21 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:22 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:22 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:22 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:22 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:22 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:23 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:23 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:23 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:23 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:23 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:24 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:24 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:24 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:24 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:24 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:25 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:25 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:25 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:25 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:25 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:26 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:26 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:26 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:26 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:27 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:27 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:27 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:27 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:27 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:28 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 4 parsl.dataflow.strategy DEBUG: Executor multi has 4 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.dflow INFO: Task 785 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 7431daaa5b5d1a710e7dfd4beffc20ca with result from task 785 parsl.dataflow.dflow DEBUG: Task 785 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/86526536-1e69-4d49-9ead-1659079add1d_calibrate_1124710_20.stdout parsl.dataflow.dflow INFO: Standard error for task 785 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/86526536-1e69-4d49-9ead-1659079add1d_calibrate_1124710_20.stderr 2024-03-07 07:34:43 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:34:43 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:43 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:43 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:43 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:43 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:44 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:44 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:44 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:44 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:44 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:44 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:45 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:45 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:45 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:45 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:45 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:45 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:45 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:46 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:46 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:46 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:46 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:46 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:46 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:47 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:47 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:47 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:47 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:47 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:47 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:47 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:48 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:50 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:50 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:50 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:50 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:52 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:52 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:52 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:52 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:52 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:52 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:53 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:53 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:53 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:53 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:53 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:53 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:53 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:54 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:54 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:54 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:54 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:54 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:54 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:54 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:55 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:55 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:55 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:55 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:55 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:55 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:56 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:56 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:56 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:56 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:56 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:56 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:56 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:57 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:57 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:57 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:57 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:57 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:57 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:58 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:34:58 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:35:13 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:35:13 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:13 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:13 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:13 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:14 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:14 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:14 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:14 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:14 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:14 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:14 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:15 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:15 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:15 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:15 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:15 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:15 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:16 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 349 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 4b68b342b8838a054aa554a61f3088a6 with result from task 349 parsl.dataflow.dflow DEBUG: Task 349 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 350 has memoization hash aa15f0a42f537f5dafb1ea7ee612dc47 parsl.dataflow.memoization INFO: Task 350 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x1535297cfce0> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 350 try 0 launched on executor multi with executor id 932 parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/e4e1b6d4-4524-4af9-8853-b9fc868936cb_calibrate_1124704_20.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/e4e1b6d4-4524-4af9-8853-b9fc868936cb_calibrate_1124704_20.stderr parsl.dataflow.dflow INFO: Standard output for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/268091b0-1476-41b3-b58e-ed35e3e78544_characterizeImage_1124704_20.stdout parsl.dataflow.dflow INFO: Standard error for task 349 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124704/268091b0-1476-41b3-b58e-ed35e3e78544_characterizeImage_1124704_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:16 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:16 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:16 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:16 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:16 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:16 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:17 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:17 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:17 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:17 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:17 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:17 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:18 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:18 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:18 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:18 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:18 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:18 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:18 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:19 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:19 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:19 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:19 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:19 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:19 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:20 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:20 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:20 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:20 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:20 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:20 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:20 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:21 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:21 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:21 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:21 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:21 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:21 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:22 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:24 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:24 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:24 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:24 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:24 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:24 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:25 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:25 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:25 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:25 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:25 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:25 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:26 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:26 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:26 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:26 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:26 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:26 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:27 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:27 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:35:47 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:35:47 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:47 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:47 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:47 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:48 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:48 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:48 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:48 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:48 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:48 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:48 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:49 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:49 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:49 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:49 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:49 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:50 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:50 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:50 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:50 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:50 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:50 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:51 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:51 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:51 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:51 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:51 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:51 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:51 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:52 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:52 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:52 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:52 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:52 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:52 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:53 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:53 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:53 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:53 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:53 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:53 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.dataflow.dflow INFO: Task 910 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry e0f20a68fb9ba4fa902a31595d51be66 with result from task 910 parsl.dataflow.dflow DEBUG: Task 910 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash 977ed98bf5be802995ebe376c454845e parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529667380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 911 try 0 launched on executor multi with executor id 933 parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/bb3a5676-adf7-4e5b-937d-0178ce4b10ad_calibrate_1124710_61.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/bb3a5676-adf7-4e5b-937d-0178ce4b10ad_calibrate_1124710_61.stderr parsl.dataflow.dflow INFO: Standard output for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/a2aadef0-9a6c-4c0d-a247-31296b9cd1c3_characterizeImage_1124710_61.stdout parsl.dataflow.dflow INFO: Standard error for task 910 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124710/a2aadef0-9a6c-4c0d-a247-31296b9cd1c3_characterizeImage_1124710_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:53 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:54 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:54 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:54 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:54 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:54 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:56 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:57 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:57 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:57 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:57 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:57 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:57 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:58 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:58 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:58 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:58 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:58 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:58 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:58 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:59 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:59 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:59 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:59 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:59 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:59 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:35:59 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:00 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:00 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:00 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:00 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:00 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:00 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:01 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:01 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:01 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 3 parsl.dataflow.strategy DEBUG: Executor multi has 3 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:36:21 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:36:21 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:21 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:21 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:22 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:22 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:22 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:22 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:22 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:22 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:23 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:23 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:23 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:23 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:23 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:23 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:23 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:24 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:24 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:24 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.dataflow.dflow INFO: Task 350 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry aa15f0a42f537f5dafb1ea7ee612dc47 with result from task 350 parsl.dataflow.dflow DEBUG: Task 350 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.dflow INFO: Standard output for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/e4e1b6d4-4524-4af9-8853-b9fc868936cb_calibrate_1124704_20.stdout parsl.dataflow.dflow INFO: Standard error for task 350 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124704/e4e1b6d4-4524-4af9-8853-b9fc868936cb_calibrate_1124704_20.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:24 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:24 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:24 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:25 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:25 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:25 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:25 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:25 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:25 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:25 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:26 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:26 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:26 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:26 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:26 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:26 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:26 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:27 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:27 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:29 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:29 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:29 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:29 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:30 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:31 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:31 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:31 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:31 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:31 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:31 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:31 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:32 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:32 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:32 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:32 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:32 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:32 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:32 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:33 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:33 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:33 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:33 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:33 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:33 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:34 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:34 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:34 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:34 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:34 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:34 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:34 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:35 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:35 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:35 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:35 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:35 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:35 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:36 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:36:56 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:36:56 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:56 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:56 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:56 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:56 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:57 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:57 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:57 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:57 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:57 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:57 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:57 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:58 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:58 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:58 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:58 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:58 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:58 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:59 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:59 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:59 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:59 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:36:59 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:00 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:00 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:00 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:00 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:00 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:00 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:02 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:03 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:03 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:03 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:03 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:03 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:03 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:04 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:04 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:04 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:04 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:04 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:04 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:04 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:05 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:05 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:05 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:05 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:05 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:05 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:06 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:06 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:06 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:06 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:06 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:07 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:07 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:07 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:07 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:07 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:07 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:07 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:08 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:08 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:08 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:08 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:08 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:08 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:09 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:09 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:09 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:09 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:09 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 911 try 0 failed parsl.dataflow.dflow INFO: Task 911 marked for retry parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/bb3a5676-adf7-4e5b-937d-0178ce4b10ad_calibrate_1124710_61.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/bb3a5676-adf7-4e5b-937d-0178ce4b10ad_calibrate_1124710_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 911 has memoization hash 977ed98bf5be802995ebe376c454845e parsl.dataflow.memoization INFO: Task 911 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x153529667380> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 911 try 1 launched on executor multi with executor id 934 parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/bb3a5676-adf7-4e5b-937d-0178ce4b10ad_calibrate_1124710_61.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/bb3a5676-adf7-4e5b-937d-0178ce4b10ad_calibrate_1124710_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:09 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:10 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:10 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:10 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:10 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:37:30 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:37:30 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:31 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:31 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:31 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:31 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:31 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:32 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:32 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:32 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:32 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:32 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:32 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:33 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:33 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:35 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:35 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:35 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:36 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:36 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:36 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:36 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:37 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:37 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:37 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:37 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:37 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:38 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:38 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:38 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:38 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:38 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:39 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:39 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:39 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:39 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:39 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:39 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:40 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:40 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:40 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:40 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:40 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:41 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:41 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:41 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:41 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:41 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:42 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:42 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:42 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:42 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:42 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:43 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:43 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:43 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:43 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:43 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:44 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:44 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:44 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:44 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:44 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:45 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:45 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:45 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:45 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:45 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:46 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:46 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:46 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:46 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:46 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:47 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:47 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:47 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:47 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:37:47 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 2 parsl.dataflow.strategy DEBUG: Executor multi has 2 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:38:03 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:38:03 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:03 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:03 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:03 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:03 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 911 try 1 failed parsl.dataflow.dflow ERROR: Task 911 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 977ed98bf5be802995ebe376c454845e with result from task 911 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Standard output for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/bb3a5676-adf7-4e5b-937d-0178ce4b10ad_calibrate_1124710_61.stdout parsl.dataflow.dflow INFO: Standard error for task 911 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124710/bb3a5676-adf7-4e5b-937d-0178ce4b10ad_calibrate_1124710_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:04 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:04 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:04 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:04 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:04 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:04 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:05 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:05 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:05 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:05 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:05 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:05 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:05 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:06 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:06 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:08 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:08 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:08 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:08 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:08 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:09 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:09 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:09 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:09 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:09 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:09 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:10 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:10 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:10 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:10 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:10 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:10 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:10 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:11 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:11 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:11 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:11 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:11 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:11 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:12 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:12 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:12 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:12 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:12 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:12 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:12 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:13 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:13 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:13 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:13 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:13 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:13 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:14 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:14 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:14 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:14 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:14 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:14 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:15 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:15 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:15 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:15 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:15 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:15 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:15 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:16 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:16 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:16 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:16 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:16 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:16 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:17 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:38:37 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:38:37 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:37 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:37 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:37 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:37 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:38 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:38 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:38 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:38 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:38 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:38 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:40 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:41 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:41 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:41 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:41 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:41 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:41 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:42 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:42 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:42 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:42 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:42 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:42 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:42 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:43 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:43 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:43 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:43 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:43 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:43 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:44 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:44 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:44 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:44 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:44 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:44 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:44 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:45 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:45 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:45 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:45 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:45 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:45 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:46 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:46 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:46 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:46 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:46 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:46 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:46 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:47 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:47 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:47 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:47 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:47 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:47 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:48 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:48 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:48 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:48 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:48 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:48 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:48 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:49 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:49 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:49 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:49 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:49 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:49 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:50 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:50 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:50 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:50 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:50 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:50 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:38:51 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:39:11 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:39:11 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:11 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:13 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:13 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:14 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:15 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:16 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:16 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:16 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:16 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:16 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:16 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:17 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:17 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:17 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:17 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:17 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:17 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:17 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:18 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:18 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:18 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:18 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:18 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:18 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:19 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:19 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:19 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:19 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:19 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:19 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:19 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:20 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:20 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:20 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:20 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:20 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:20 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:21 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:21 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:21 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:21 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:21 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:21 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:22 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:22 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:22 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:22 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:22 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:22 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:22 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:23 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:23 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:23 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:23 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:23 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:23 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:23 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:24 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:24 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:24 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:24 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:24 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:24 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:25 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:25 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:25 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:25 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:25 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:25 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:25 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:26 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:26 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:26 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:26 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:26 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:26 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:39:42 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:39:42 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:42 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:42 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:42 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:42 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:42 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:43 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:43 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:43 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:43 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:43 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:43 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:43 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:44 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:44 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:44 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:44 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:44 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:44 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:45 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:45 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:45 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:45 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:45 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:45 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:47 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:48 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:48 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:48 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:49 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:49 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:50 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:50 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:50 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:50 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:50 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:50 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:51 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:51 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:51 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:51 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:51 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:51 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:52 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:52 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:52 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:52 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:52 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:52 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:52 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:53 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:53 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:53 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:53 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:53 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:53 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:54 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:54 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:54 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:54 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:54 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:54 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:54 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:55 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:55 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:55 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:55 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:55 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:55 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:56 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:56 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:56 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:56 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:56 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:56 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:56 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:39:57 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887808 R 16887809 R 16887810 R 16887811 R 16887812 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:40:12 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:40:12 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:12 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:12 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:12 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:12 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:13 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:13 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:13 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:13 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:13 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:13 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:14 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:14 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:14 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:14 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:14 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:14 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:15 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:15 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:15 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:15 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:15 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:15 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:15 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:16 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:16 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:16 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:16 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:16 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:16 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:17 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:17 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:17 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:17 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:17 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:17 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:18 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:18 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:18 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:18 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:18 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:18 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:19 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:19 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:19 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:19 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:19 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:19 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:21 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:22 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:22 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:23 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:24 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:24 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:24 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:24 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:24 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:24 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:25 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:25 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:25 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:25 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:25 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:25 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:25 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:26 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:26 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:26 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:26 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:26 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:26 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:26 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:27 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:27 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:27 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:27 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:27 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:40:42 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:40:42 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:43 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:43 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:43 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:43 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:43 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:43 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:44 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:44 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:44 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:44 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:44 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:44 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:45 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:45 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:45 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:45 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:45 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:45 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:45 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:46 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:46 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:46 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:46 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:46 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:46 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:47 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:47 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:47 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:47 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:47 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:47 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:47 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:48 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:48 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:48 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:48 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:48 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:48 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:48 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:49 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:49 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:49 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:49 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:49 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:49 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:50 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:50 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:50 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:50 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:50 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:50 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:50 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:51 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:51 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:51 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:51 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:51 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:51 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:52 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:52 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:52 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:52 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:52 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:52 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:52 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:53 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:53 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:53 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:53 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:53 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:53 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:55 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:56 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:56 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:56 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:40:56 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 CG 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status CG to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:41:16 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:41:16 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:16 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:17 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:17 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:17 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:17 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:17 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:17 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:18 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:18 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:18 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:18 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:18 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:18 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:18 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:19 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:19 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:19 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:19 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:19 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:19 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:20 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:20 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:20 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:20 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:20 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:20 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:20 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:21 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:21 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:21 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:21 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:21 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:21 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:22 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:22 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:22 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:22 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:22 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:22 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:22 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:23 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:23 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:23 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:23 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:23 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:23 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:24 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:24 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:24 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:24 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:24 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:24 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:24 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:25 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:25 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:25 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:25 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:25 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:25 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:26 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:26 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:26 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:26 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:28 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:28 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:28 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:28 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:29 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:29 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:29 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:29 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:29 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:29 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:30 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:30 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:30 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:41:50 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:41:50 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:50 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:50 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:51 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:51 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:51 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:51 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:51 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:51 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:52 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:52 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:52 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:52 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:52 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:52 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:52 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:53 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:53 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:53 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:53 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:53 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:53 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:54 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:54 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:54 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:54 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:54 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:54 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:54 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:55 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:55 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:55 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:55 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:55 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:55 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:56 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:56 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:56 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:56 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:56 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:56 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:56 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:57 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:57 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:57 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:57 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:57 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:57 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:58 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:58 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:58 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:58 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:58 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:58 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:58 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:41:59 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:01 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:01 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:01 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:01 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:01 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:01 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:02 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:02 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:02 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:02 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:02 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:02 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:02 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:03 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:03 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:03 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:03 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:03 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:03 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:04 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:04 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:42:24 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:42:24 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:24 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:24 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:24 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:25 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:25 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:25 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:25 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:25 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:25 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:25 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:26 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:26 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:26 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:26 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:26 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:26 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:27 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:27 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:27 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:27 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:27 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:27 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:27 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:28 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:28 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:28 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:28 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:28 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:28 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:29 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:29 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:29 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:29 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:29 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:29 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:29 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:30 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:30 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:30 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:30 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:30 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:30 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:31 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:31 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:31 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:31 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:31 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:33 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:33 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:34 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:34 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:34 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:34 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:34 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:34 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:34 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:35 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:35 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:35 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:35 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:35 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:35 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:36 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:36 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:36 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:36 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:36 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:36 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:36 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:37 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:37 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:37 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:37 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:37 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:37 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:38 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:42:58 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:42:58 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:58 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:58 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:58 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:58 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:59 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:59 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:59 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:59 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:59 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:59 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:42:59 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:00 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:00 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:00 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:00 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:00 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:00 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:01 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:01 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:01 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:01 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:01 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:01 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:01 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:02 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:02 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:02 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:02 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:02 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:02 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:03 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:03 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:03 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:03 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:03 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:03 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:03 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:04 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:04 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:06 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:06 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:06 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:06 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:07 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:07 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:07 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:07 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:07 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:07 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:07 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:08 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:08 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:08 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:08 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:08 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:08 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:09 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:09 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:09 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:09 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:09 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:09 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:09 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:10 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:10 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:10 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:10 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:10 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:10 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:11 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:11 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:11 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:11 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:11 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:11 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:11 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 PD 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status PD to parsl state JobState.PENDING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 77/1 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:43:32 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:43:32 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:32 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:32 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:32 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:32 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:32 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:33 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:33 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:33 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:33 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:33 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:33 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:34 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:34 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:34 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:34 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:34 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:34 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:34 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:35 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:35 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:35 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:35 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:35 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:35 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:35 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:36 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:36 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:36 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:36 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:36 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:36 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:39 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:39 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:39 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:39 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:39 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:39 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:40 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:40 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:40 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:40 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:40 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:40 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:41 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:41 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:41 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:41 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:41 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:41 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:41 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:42 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:42 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:42 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:42 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:42 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:42 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:43 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:43 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:43 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:43 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:43 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:43 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:44 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:44 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:44 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:44 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:44 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:44 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:45 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:45 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:45 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:45 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:45 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:45 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:45 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:43:46 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 225 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:44:06 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:44:06 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:06 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:06 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:06 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:06 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:07 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:07 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:07 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:07 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:07 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:07 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:08 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:08 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:08 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:08 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:08 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:08 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:09 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:09 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:09 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:09 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:09 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:11 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:11 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:12 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:12 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:12 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:13 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:13 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:13 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:13 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:13 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:13 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:13 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:14 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:14 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:14 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:14 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:14 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:14 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:15 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:15 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:15 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:15 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:15 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:15 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:15 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:16 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:16 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:16 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:16 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:16 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:16 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:16 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:17 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:17 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:17 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:17 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:17 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:17 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:18 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:18 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:18 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:18 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:18 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:18 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:18 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:19 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:19 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:19 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:19 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:19 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:19 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:20 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:20 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:20 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:20 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:44:40 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:44:40 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:40 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:41 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:41 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:41 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:41 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:41 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:41 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:41 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:42 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:42 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:42 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:42 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:42 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:44 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:45 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:45 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:45 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:45 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:45 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:46 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:46 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:46 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:46 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:46 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:46 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:46 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:47 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:47 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:47 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:47 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:47 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:47 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:47 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:48 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:48 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:48 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:48 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:48 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:48 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:49 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:49 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:49 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:49 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:49 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:49 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:49 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:50 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:50 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:50 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:50 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:50 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:50 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:50 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:51 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:51 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:51 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:51 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:51 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:51 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:52 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:52 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:52 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:52 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:52 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:52 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:52 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:53 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:53 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:53 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:53 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:53 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:53 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:54 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:54 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:54 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:44:54 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:45:14 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:45:14 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:14 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:15 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:15 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:15 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:15 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:15 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:17 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:18 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:18 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:18 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:18 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:18 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:18 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:19 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:19 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:19 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:19 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:19 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:19 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:19 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:20 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:20 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:20 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:20 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:20 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:21 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:21 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:21 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:21 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:21 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:21 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:22 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:22 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:22 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:22 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:22 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:22 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:22 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:23 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:23 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:23 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:23 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:23 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:23 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:23 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:24 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:24 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:24 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:24 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:24 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:24 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:25 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:25 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:25 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:25 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:25 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:25 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:25 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:26 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:26 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:26 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:26 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:26 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:26 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:27 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:27 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:27 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:27 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:27 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:27 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:27 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:28 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:28 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:28 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:28 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:28 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:45:48 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:45:48 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:50 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:50 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:50 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:50 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:50 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:51 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:51 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:51 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:51 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:51 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:51 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:51 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:52 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:52 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:52 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:52 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:52 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:52 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:53 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:53 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:53 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:53 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:53 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:53 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:53 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:54 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:54 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:54 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:54 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:54 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:54 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:55 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:55 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:55 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:55 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:55 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:55 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:55 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:56 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:56 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:56 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:56 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:56 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:56 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:57 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:57 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:57 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:57 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:57 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:57 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:57 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:58 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:58 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:58 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:58 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:58 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:58 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:58 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:59 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:59 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:59 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:59 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:59 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:45:59 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:00 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:00 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:00 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:00 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:00 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:00 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:00 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:01 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:01 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:01 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:01 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:01 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.dflow INFO: Task 748 completed (launched -> exec_done) parsl.dataflow.memoization DEBUG: Storing app cache entry 9f59ee1ddb3f26fadbccfa6d29e6533d with result from task 748 parsl.dataflow.dflow DEBUG: Task 748 checkpointed parsl.dataflow.dflow INFO: Done checkpointing 1 tasks parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash 5195410a92dac4897fb5f5773c7dfb05 parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 0 launched on executor multi with executor id 935 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d2a14d49-dcd8-4d2b-ab61-0aeb90686630_calibrate_1124707_61.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d2a14d49-dcd8-4d2b-ab61-0aeb90686630_calibrate_1124707_61.stderr parsl.dataflow.dflow INFO: Standard output for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e546f862-4a02-4e0f-b4b3-bed4a3b43471_characterizeImage_1124707_61.stdout parsl.dataflow.dflow INFO: Standard error for task 748 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/characterizeImage/1124707/e546f862-4a02-4e0f-b4b3-bed4a3b43471_characterizeImage_1124707_61.stderr parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:46:21 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:46:21 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:22 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:22 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:22 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:22 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:22 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:23 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:23 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:23 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:23 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:24 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:24 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:24 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:24 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:24 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:25 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:25 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:25 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:25 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:25 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:26 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:26 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:26 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:26 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:26 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:27 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:27 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:27 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:27 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:27 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:28 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:28 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:28 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:28 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:29 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:29 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:29 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:29 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:29 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:30 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:30 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:30 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:30 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:30 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:31 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:31 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:31 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:31 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:31 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:32 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:32 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:32 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:32 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:33 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:33 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:33 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:33 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:33 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:34 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:34 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:34 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:34 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:34 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:35 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:35 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:35 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:35 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:35 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:36 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:36 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:36 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:36 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:36 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:37 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:37 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:37 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:37 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:46:52 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:46:52 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:54 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:54 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:54 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:54 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:55 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:55 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:55 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:55 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:55 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:56 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:56 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:56 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:56 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:56 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:57 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:57 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:57 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:57 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:58 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:58 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:58 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:58 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:58 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:59 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:59 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:59 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:59 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:46:59 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:00 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:00 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:00 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:00 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:00 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:01 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:01 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:01 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:01 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:02 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:02 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:02 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:02 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:02 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:03 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:03 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:03 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:03 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:03 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:04 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:04 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:04 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:04 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:04 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:05 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:05 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:05 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:05 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:05 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:06 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:06 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:06 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:06 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:06 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:07 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:07 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:07 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:07 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:07 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:08 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:08 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:08 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:08 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:08 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:09 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:09 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 749 try 0 failed parsl.dataflow.dflow INFO: Task 749 marked for retry parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d2a14d49-dcd8-4d2b-ab61-0aeb90686630_calibrate_1124707_61.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d2a14d49-dcd8-4d2b-ab61-0aeb90686630_calibrate_1124707_61.stderr parsl.dataflow.memoization DEBUG: Ignoring these kwargs for checkpointing: ['stderr', 'stdout'] parsl.dataflow.memoization DEBUG: Ignoring kwarg stderr parsl.dataflow.memoization DEBUG: Ignoring kwarg stdout parsl.dataflow.memoization DEBUG: Task 749 has memoization hash 5195410a92dac4897fb5f5773c7dfb05 parsl.dataflow.memoization INFO: Task 749 had no result in cache parsl.executors.high_throughput.executor DEBUG: Pushing function .wrapper at 0x15352977da80> to queue with args ("'${CTRL_MPEXEC_DIR}/bin/pipetask --long-log --log-level=VERBOSE run-qbb /mmfs1/home/stevengs/dirac/D...",) parsl.dataflow.dflow INFO: Parsl task 749 try 1 launched on executor multi with executor id 936 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d2a14d49-dcd8-4d2b-ab61-0aeb90686630_calibrate_1124707_61.stdout parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d2a14d49-dcd8-4d2b-ab61-0aeb90686630_calibrate_1124707_61.stderr parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:09 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:09 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:47:24 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:47:24 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:26 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:26 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:26 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:26 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:26 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:27 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:27 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:27 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:27 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:27 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:27 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:27 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:28 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:28 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:28 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:28 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:28 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:28 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:28 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:29 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:29 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:29 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:29 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:29 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:29 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:30 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:30 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:30 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:30 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:30 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:30 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:30 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:31 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:31 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:31 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:31 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:31 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:31 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:32 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:32 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:32 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:32 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:32 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:32 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:32 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:33 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:33 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:33 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:33 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:33 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:33 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:34 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:34 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:34 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:34 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:34 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:34 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:34 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:35 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:35 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:35 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:35 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:35 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:35 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:35 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:36 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:36 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:36 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:36 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:36 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:36 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:37 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:37 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:37 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:37 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:37 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 2024-03-07 07:47:57 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:47:57 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:58 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:58 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:47:59 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:00 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:00 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:00 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:01 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:01 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:01 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:01 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:01 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:01 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:01 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:02 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:02 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:02 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:02 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:02 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:02 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:03 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:03 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:03 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:03 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:03 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:03 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:03 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:04 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:04 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:04 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887749,16887750,16887751,16887752,16887753,16887754,16887755,16887756,16887757,16887758,16887759,16887760,16887761,16887762,16887763,16887766,16887767,16887768,16887769,16887770,16887771,16887772,16887773,16887774,16887775,16887776,16887777,16887778,16887779,16887780' parsl.providers.slurm.slurm DEBUG: squeue returned 16887775 R 16887776 R 16887777 R 16887778 R 16887779 R 16887780 R 16887773 R 16887774 R 16887769 R 16887770 R 16887771 R 16887772 R 16887766 R 16887767 R 16887768 R 16887762 R 16887763 R 16887761 R 16887759 R 16887760 R 16887757 R 16887758 R 16887752 R 16887753 R 16887754 R 16887755 R 16887756 R 16887749 R 16887750 R 16887751 R parsl.providers.slurm.slurm DEBUG: Updating job 16887775 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887776 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887777 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887778 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887779 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887780 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887773 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887774 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887769 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887770 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887771 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887772 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887766 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887767 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887768 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887762 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887763 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887761 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887759 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887760 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887757 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887758 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887752 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887753 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887754 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887755 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887756 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887749 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887750 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887751 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:04 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:04 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:04 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:05 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:05 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887781,16887782,16887783,16887784,16887785' parsl.providers.slurm.slurm DEBUG: squeue returned 16887784 R 16887785 R 16887783 R 16887782 R 16887781 R parsl.providers.slurm.slurm DEBUG: Updating job 16887784 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887785 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887783 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887782 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887781 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:05 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:05 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:05 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:05 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:05 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:06 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:06 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:06 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:06 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:06 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:06 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:06 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:07 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:07 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:07 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:07 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:07 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:07 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:08 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:08 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887786,16887787,16887788,16887789,16887790,16887791,16887792,16887793,16887794,16887795,16887796,16887797,16887798,16887799,16887800,16887801,16887802,16887803,16887804,16887805' parsl.providers.slurm.slurm DEBUG: squeue returned 16887804 R 16887805 R 16887800 R 16887801 R 16887802 R 16887803 R 16887796 R 16887797 R 16887798 R 16887799 R 16887791 R 16887792 R 16887793 R 16887794 R 16887795 R 16887789 R 16887790 R 16887788 R 16887786 R 16887787 R parsl.providers.slurm.slurm DEBUG: Updating job 16887804 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887805 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887800 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887801 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887802 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887803 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887796 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887797 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887798 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887799 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887791 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887792 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887793 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887794 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887795 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887789 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887790 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887788 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887786 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887787 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:08 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:08 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:08 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:08 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:08 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:09 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:09 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:09 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:09 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:09 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:09 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:10 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:10 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:10 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:10 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:10 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:10 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:10 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:11 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:11 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:11 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING 2024-03-07 07:48:11 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint parsl.providers.slurm.slurm DEBUG: Executing squeue --noheader --format='%i %t' --job '16887806,16887807,16887808,16887809,16887810,16887811,16887812,16887813,16887814,16887815,16887816,16887817,16887818,16887819,16887820,16887821,16887822,16887823,16887824,16887825,16887826,16887827' parsl.providers.slurm.slurm DEBUG: squeue returned 16887812 R 16887808 R 16887809 R 16887810 R 16887811 R 16887813 R 16887814 R 16887815 R 16887816 R 16887817 R 16887818 R 16887819 R 16887820 R 16887821 R 16887822 R 16887823 R 16887824 R 16887825 R 16887826 R 16887827 R 16887806 R 16887807 R parsl.providers.slurm.slurm DEBUG: Updating job 16887812 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887808 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887809 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887810 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887811 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887813 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887814 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887815 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887816 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887817 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887818 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887819 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887820 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887821 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887822 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887823 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887824 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887825 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887826 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887827 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887806 with slurm status R to parsl state JobState.RUNNING parsl.providers.slurm.slurm DEBUG: Updating job 16887807 with slurm status R to parsl state JobState.RUNNING parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.dataflow.strategy DEBUG: general strategy starting with strategy_type simple for 1 executors parsl.dataflow.strategy DEBUG: Strategizing for executor multi parsl.dataflow.strategy DEBUG: Slot ratio calculation: active_slots = 78, active_tasks = 1 parsl.dataflow.strategy DEBUG: Executor multi has 1 active tasks, 78/0 running/pending blocks, and 233 connected workers parsl.dataflow.strategy DEBUG: Strategy case 4b: more slots than tasks parsl.dataflow.strategy DEBUG: This strategy does not support scaling in parsl.process_loggers DEBUG: Normal ending for _general_strategy on thread JobStatusPoller-Timer-Thread-23318106354576 parsl.app.errors DEBUG: Reraising exception of type parsl.dataflow.dflow DEBUG: Task 749 try 1 failed parsl.dataflow.dflow ERROR: Task 749 failed after 1 retry attempts Traceback (most recent call last): File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 300, in handle_exec_update res = self._unwrap_remote_exception_wrapper(future) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/dataflow/dflow.py", line 566, in _unwrap_remote_exception_wrapper result.reraise() File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 123, in reraise reraise(t, v, v.__traceback__) File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/six.py", line 719, in reraise raise value File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/errors.py", line 146, in wrapper return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^ File "/mmfs1/gscratch/dirac/shared/opt/conda/envs/lsst-scipipe-8.0.0/lib/python3.11/site-packages/parsl/app/bash.py", line 86, in remote_side_bash_executor raise pe.BashExitFailure(func_name, proc.returncode) ^^^^^^^^^^^^^^^^^ parsl.app.errors.BashExitFailure: bash_app calibrate failed with unix exit code 1 parsl.dataflow.memoization DEBUG: Storing app cache entry 5195410a92dac4897fb5f5773c7dfb05 with result from task 749 parsl.dataflow.dflow INFO: DFK cleanup initiated parsl.dataflow.dflow INFO: Summary of tasks in DFK: parsl.dataflow.dflow INFO: Tasks in state States.unsched: 0 parsl.dataflow.dflow INFO: Tasks in state States.pending: 0 parsl.dataflow.dflow INFO: Tasks in state States.running: 0 parsl.dataflow.dflow INFO: Tasks in state States.exec_done: 910 parsl.dataflow.dflow INFO: Tasks in state States.failed: 13 parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Tasks in state States.dep_fail: 7 parsl.dataflow.dflow INFO: Standard output for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d2a14d49-dcd8-4d2b-ab61-0aeb90686630_calibrate_1124707_61.stdout parsl.dataflow.dflow INFO: Tasks in state States.launched: 0 parsl.dataflow.dflow INFO: Standard error for task 749 available at /mmfs1/home/stevengs/dirac/DEEP/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/submit/DEEP/20220823/B1d/science#step1/20240307T072452Z/logs/calibrate/1124707/d2a14d49-dcd8-4d2b-ab61-0aeb90686630_calibrate_1124707_61.stderr parsl.dataflow.dflow INFO: Tasks in state States.fail_retryable: 0 parsl.dataflow.dflow INFO: Tasks in state States.memo_done: 0 parsl.dataflow.dflow INFO: Tasks in state States.joining: 0 parsl.dataflow.dflow INFO: Tasks in state States.running_ended: 0 parsl.dataflow.dflow INFO: End of summary parsl.dataflow.dflow DEBUG: No tasks checkpointed in this pass. parsl.dataflow.dflow INFO: Closing job status poller parsl.dataflow.dflow INFO: Terminated job status poller parsl.dataflow.dflow INFO: Scaling in and shutting down executors parsl.dataflow.dflow INFO: Scaling in executor multi parsl.executors.high_throughput.executor DEBUG: Scale in called, blocks=78, block_ids=[] parsl.executors.high_throughput.executor DEBUG: Scale in selecting from 78 blocks parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e570737aa114 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e570737aa114 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 059c01b599fc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 059c01b599fc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4e73ff5ebf48 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4e73ff5ebf48 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e8340c495868 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e8340c495868 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2ec876455125 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2ec876455125 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1bc0b5a3dc9d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1bc0b5a3dc9d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bfa5ca26fa2f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bfa5ca26fa2f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 943a84b7c592 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 943a84b7c592 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fa618d52f57d parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fa618d52f57d parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 09ba5b28579e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 09ba5b28579e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 608e5b6c5a39 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 608e5b6c5a39 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 447dc33881ce parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 447dc33881ce parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ffd22f90bee4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ffd22f90bee4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0faee3e312cf parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0faee3e312cf parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 174a1687b7aa parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 174a1687b7aa parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 30ab1ce03cbc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 30ab1ce03cbc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 950654412984 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 950654412984 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 96281108225c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 96281108225c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 91e87c57e3ef parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 91e87c57e3ef parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 59c8a2a4b787 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 59c8a2a4b787 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: b5521f5f1d38 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: b5521f5f1d38 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ba8b1995bdc5 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ba8b1995bdc5 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: faaed070e449 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: faaed070e449 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf0ae45f81ff parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf0ae45f81ff parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: fb4f779e48e2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: fb4f779e48e2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2b80b472dbb1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2b80b472dbb1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d8475258acea parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d8475258acea parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8cfc2d18f296 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8cfc2d18f296 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d9db7c4061a9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d9db7c4061a9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 17a1bd79707a parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 17a1bd79707a parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d7665306fe34 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d7665306fe34 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4ce0097b6b4e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4ce0097b6b4e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bf4c37cbc2f7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bf4c37cbc2f7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2a48430d5ea7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2a48430d5ea7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bdbac1c6b627 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bdbac1c6b627 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f1045950ae01 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f1045950ae01 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f2d9070ff4e2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f2d9070ff4e2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 8aa70b0ee3b9 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 8aa70b0ee3b9 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1e516a9a12ee parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1e516a9a12ee parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e0aba6a3d153 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e0aba6a3d153 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 68c97a7dd346 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 68c97a7dd346 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 3a1d06892cdc parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 3a1d06892cdc parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 574835aef136 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 574835aef136 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0b2cf660d6df parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0b2cf660d6df parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 30b42a4715c2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 30b42a4715c2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 38c38dd20fc4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 38c38dd20fc4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: c94005816feb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: c94005816feb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f0865ab55a02 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f0865ab55a02 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 2998037e0df1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 2998037e0df1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 74c2c579b61f parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 74c2c579b61f parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 6d976b0173ea parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 6d976b0173ea parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: f2e0b450bbce parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: f2e0b450bbce parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 53cc97be6264 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 53cc97be6264 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 28a2ef276e4b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 28a2ef276e4b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 22eae19862cd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 22eae19862cd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0d7b503ce1ba parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0d7b503ce1ba parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 984aaa2a38cb parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 984aaa2a38cb parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5ddafe40ff25 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5ddafe40ff25 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ecc3fc1a4f44 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ecc3fc1a4f44 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 4c4da6ae77a7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 4c4da6ae77a7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1d9326944ec4 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1d9326944ec4 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d2fc7f43b4be parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d2fc7f43b4be parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 01da8171298b parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 01da8171298b parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: ca04fb638d88 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: ca04fb638d88 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: bbdfa44a9560 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: bbdfa44a9560 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9311a94c2963 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9311a94c2963 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 98e47413cebd parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 98e47413cebd parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e228ec6a6dd2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e228ec6a6dd2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 396c08aa242c parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 396c08aa242c parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9f9a773d62d7 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9f9a773d62d7 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: d65e7ec1eb6e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: d65e7ec1eb6e parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 076e14105bf1 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 076e14105bf1 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 887c8bde4a78 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 887c8bde4a78 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: e264d7dc4030 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: e264d7dc4030 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 5737245d8938 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 5737245d8938 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 9dba60a395b2 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 9dba60a395b2 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 1c385fbfa584 parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 1c385fbfa584 parsl.executors.high_throughput.executor DEBUG: Sending hold to manager: 0253ddbe480e parsl.executors.high_throughput.executor DEBUG: Sent hold request to manager: 0253ddbe480e 2024-03-07 07:48:16 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 2024-03-07 07:48:16 proc_lsst.multi:201 [INFO] cancelling 16887793 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887793 on provider escience_a40 2024-03-07 07:48:16 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro 2024-03-07 07:48:16 proc_lsst.multi:201 [INFO] cancelling 16887755 on provider astro proc_lsst.multi INFO: cancelling 16887755 on provider astro 2024-03-07 07:48:16 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 2024-03-07 07:48:16 proc_lsst.multi:201 [INFO] cancelling 16887801 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887801 on provider escience_a40 2024-03-07 07:48:16 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint 2024-03-07 07:48:16 proc_lsst.multi:201 [INFO] cancelling 16887812 on provider checkpoint proc_lsst.multi INFO: cancelling 16887812 on provider checkpoint 2024-03-07 07:48:16 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro 2024-03-07 07:48:16 proc_lsst.multi:201 [INFO] cancelling 16887761 on provider astro proc_lsst.multi INFO: cancelling 16887761 on provider astro 2024-03-07 07:48:16 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro 2024-03-07 07:48:16 proc_lsst.multi:201 [INFO] cancelling 16887772 on provider astro proc_lsst.multi INFO: cancelling 16887772 on provider astro 2024-03-07 07:48:16 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 2024-03-07 07:48:16 proc_lsst.multi:201 [INFO] cancelling 16887804 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887804 on provider escience_a40 2024-03-07 07:48:17 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro 2024-03-07 07:48:17 proc_lsst.multi:201 [INFO] cancelling 16887752 on provider astro proc_lsst.multi INFO: cancelling 16887752 on provider astro 2024-03-07 07:48:17 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint 2024-03-07 07:48:17 proc_lsst.multi:201 [INFO] cancelling 16887825 on provider checkpoint proc_lsst.multi INFO: cancelling 16887825 on provider checkpoint 2024-03-07 07:48:17 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k 2024-03-07 07:48:17 proc_lsst.multi:201 [INFO] cancelling 16887782 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16887782 on provider escience_rtx6k 2024-03-07 07:48:17 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint 2024-03-07 07:48:17 proc_lsst.multi:201 [INFO] cancelling 16887806 on provider checkpoint proc_lsst.multi INFO: cancelling 16887806 on provider checkpoint 2024-03-07 07:48:17 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 2024-03-07 07:48:17 proc_lsst.multi:201 [INFO] cancelling 16887800 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887800 on provider escience_a40 2024-03-07 07:48:17 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro 2024-03-07 07:48:17 proc_lsst.multi:201 [INFO] cancelling 16887767 on provider astro proc_lsst.multi INFO: cancelling 16887767 on provider astro 2024-03-07 07:48:17 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint 2024-03-07 07:48:17 proc_lsst.multi:201 [INFO] cancelling 16887824 on provider checkpoint proc_lsst.multi INFO: cancelling 16887824 on provider checkpoint 2024-03-07 07:48:17 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint 2024-03-07 07:48:17 proc_lsst.multi:201 [INFO] cancelling 16887815 on provider checkpoint proc_lsst.multi INFO: cancelling 16887815 on provider checkpoint 2024-03-07 07:48:17 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint 2024-03-07 07:48:18 proc_lsst.multi:201 [INFO] cancelling 16887822 on provider checkpoint proc_lsst.multi INFO: cancelling 16887822 on provider checkpoint 2024-03-07 07:48:18 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint 2024-03-07 07:48:18 proc_lsst.multi:201 [INFO] cancelling 16887827 on provider checkpoint proc_lsst.multi INFO: cancelling 16887827 on provider checkpoint 2024-03-07 07:48:18 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro 2024-03-07 07:48:18 proc_lsst.multi:201 [INFO] cancelling 16887763 on provider astro proc_lsst.multi INFO: cancelling 16887763 on provider astro 2024-03-07 07:48:18 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k 2024-03-07 07:48:18 proc_lsst.multi:201 [INFO] cancelling 16887785 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16887785 on provider escience_rtx6k 2024-03-07 07:48:18 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint 2024-03-07 07:48:18 proc_lsst.multi:201 [INFO] cancelling 16887808 on provider checkpoint proc_lsst.multi INFO: cancelling 16887808 on provider checkpoint 2024-03-07 07:48:18 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro 2024-03-07 07:48:18 proc_lsst.multi:201 [INFO] cancelling 16887776 on provider astro proc_lsst.multi INFO: cancelling 16887776 on provider astro 2024-03-07 07:48:18 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint 2024-03-07 07:48:18 proc_lsst.multi:201 [INFO] cancelling 16887820 on provider checkpoint proc_lsst.multi INFO: cancelling 16887820 on provider checkpoint 2024-03-07 07:48:18 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro 2024-03-07 07:48:18 proc_lsst.multi:201 [INFO] cancelling 16887769 on provider astro proc_lsst.multi INFO: cancelling 16887769 on provider astro 2024-03-07 07:48:18 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro 2024-03-07 07:48:18 proc_lsst.multi:201 [INFO] cancelling 16887778 on provider astro proc_lsst.multi INFO: cancelling 16887778 on provider astro 2024-03-07 07:48:18 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k 2024-03-07 07:48:18 proc_lsst.multi:201 [INFO] cancelling 16887783 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16887783 on provider escience_rtx6k 2024-03-07 07:48:19 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 2024-03-07 07:48:19 proc_lsst.multi:201 [INFO] cancelling 16887795 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887795 on provider escience_a40 2024-03-07 07:48:19 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro 2024-03-07 07:48:19 proc_lsst.multi:201 [INFO] cancelling 16887771 on provider astro proc_lsst.multi INFO: cancelling 16887771 on provider astro 2024-03-07 07:48:19 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint 2024-03-07 07:48:19 proc_lsst.multi:201 [INFO] cancelling 16887807 on provider checkpoint proc_lsst.multi INFO: cancelling 16887807 on provider checkpoint 2024-03-07 07:48:19 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k 2024-03-07 07:48:19 proc_lsst.multi:201 [INFO] cancelling 16887784 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16887784 on provider escience_rtx6k 2024-03-07 07:48:19 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 2024-03-07 07:48:19 proc_lsst.multi:201 [INFO] cancelling 16887787 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887787 on provider escience_a40 2024-03-07 07:48:19 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro 2024-03-07 07:48:19 proc_lsst.multi:201 [INFO] cancelling 16887780 on provider astro proc_lsst.multi INFO: cancelling 16887780 on provider astro 2024-03-07 07:48:19 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint 2024-03-07 07:48:19 proc_lsst.multi:201 [INFO] cancelling 16887823 on provider checkpoint proc_lsst.multi INFO: cancelling 16887823 on provider checkpoint 2024-03-07 07:48:19 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint 2024-03-07 07:48:19 proc_lsst.multi:201 [INFO] cancelling 16887819 on provider checkpoint proc_lsst.multi INFO: cancelling 16887819 on provider checkpoint 2024-03-07 07:48:19 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 2024-03-07 07:48:19 proc_lsst.multi:201 [INFO] cancelling 16887802 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887802 on provider escience_a40 2024-03-07 07:48:20 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint 2024-03-07 07:48:20 proc_lsst.multi:201 [INFO] cancelling 16887816 on provider checkpoint proc_lsst.multi INFO: cancelling 16887816 on provider checkpoint 2024-03-07 07:48:20 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint 2024-03-07 07:48:20 proc_lsst.multi:201 [INFO] cancelling 16887811 on provider checkpoint proc_lsst.multi INFO: cancelling 16887811 on provider checkpoint 2024-03-07 07:48:20 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint 2024-03-07 07:48:20 proc_lsst.multi:201 [INFO] cancelling 16887813 on provider checkpoint proc_lsst.multi INFO: cancelling 16887813 on provider checkpoint 2024-03-07 07:48:20 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint 2024-03-07 07:48:20 proc_lsst.multi:201 [INFO] cancelling 16887809 on provider checkpoint proc_lsst.multi INFO: cancelling 16887809 on provider checkpoint 2024-03-07 07:48:20 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro 2024-03-07 07:48:20 proc_lsst.multi:201 [INFO] cancelling 16887759 on provider astro proc_lsst.multi INFO: cancelling 16887759 on provider astro 2024-03-07 07:48:20 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro 2024-03-07 07:48:20 proc_lsst.multi:201 [INFO] cancelling 16887774 on provider astro proc_lsst.multi INFO: cancelling 16887774 on provider astro 2024-03-07 07:48:20 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint 2024-03-07 07:48:20 proc_lsst.multi:201 [INFO] cancelling 16887817 on provider checkpoint proc_lsst.multi INFO: cancelling 16887817 on provider checkpoint 2024-03-07 07:48:21 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint 2024-03-07 07:48:21 proc_lsst.multi:201 [INFO] cancelling 16887821 on provider checkpoint proc_lsst.multi INFO: cancelling 16887821 on provider checkpoint 2024-03-07 07:48:21 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro 2024-03-07 07:48:21 proc_lsst.multi:201 [INFO] cancelling 16887773 on provider astro proc_lsst.multi INFO: cancelling 16887773 on provider astro 2024-03-07 07:48:21 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro 2024-03-07 07:48:21 proc_lsst.multi:201 [INFO] cancelling 16887766 on provider astro proc_lsst.multi INFO: cancelling 16887766 on provider astro 2024-03-07 07:48:21 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k 2024-03-07 07:48:21 proc_lsst.multi:201 [INFO] cancelling 16887781 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16887781 on provider escience_rtx6k 2024-03-07 07:48:21 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint 2024-03-07 07:48:21 proc_lsst.multi:201 [INFO] cancelling 16887818 on provider checkpoint proc_lsst.multi INFO: cancelling 16887818 on provider checkpoint 2024-03-07 07:48:21 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 2024-03-07 07:48:21 proc_lsst.multi:201 [INFO] cancelling 16887803 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887803 on provider escience_a40 2024-03-07 07:48:21 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro 2024-03-07 07:48:21 proc_lsst.multi:201 [INFO] cancelling 16887775 on provider astro proc_lsst.multi INFO: cancelling 16887775 on provider astro 2024-03-07 07:48:21 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro 2024-03-07 07:48:21 proc_lsst.multi:201 [INFO] cancelling 16887777 on provider astro proc_lsst.multi INFO: cancelling 16887777 on provider astro 2024-03-07 07:48:22 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 2024-03-07 07:48:22 proc_lsst.multi:201 [INFO] cancelling 16887794 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887794 on provider escience_a40 2024-03-07 07:48:22 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint 2024-03-07 07:48:22 proc_lsst.multi:201 [INFO] cancelling 16887826 on provider checkpoint proc_lsst.multi INFO: cancelling 16887826 on provider checkpoint 2024-03-07 07:48:22 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 2024-03-07 07:48:22 proc_lsst.multi:201 [INFO] cancelling 16887788 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887788 on provider escience_a40 2024-03-07 07:48:22 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 2024-03-07 07:48:22 proc_lsst.multi:201 [INFO] cancelling 16887792 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887792 on provider escience_a40 2024-03-07 07:48:22 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro 2024-03-07 07:48:22 proc_lsst.multi:201 [INFO] cancelling 16887749 on provider astro proc_lsst.multi INFO: cancelling 16887749 on provider astro 2024-03-07 07:48:22 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro 2024-03-07 07:48:22 proc_lsst.multi:201 [INFO] cancelling 16887758 on provider astro proc_lsst.multi INFO: cancelling 16887758 on provider astro 2024-03-07 07:48:22 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro 2024-03-07 07:48:22 proc_lsst.multi:201 [INFO] cancelling 16887750 on provider astro proc_lsst.multi INFO: cancelling 16887750 on provider astro 2024-03-07 07:48:22 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 2024-03-07 07:48:22 proc_lsst.multi:201 [INFO] cancelling 16887798 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887798 on provider escience_a40 2024-03-07 07:48:23 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint 2024-03-07 07:48:23 proc_lsst.multi:201 [INFO] cancelling 16887814 on provider checkpoint proc_lsst.multi INFO: cancelling 16887814 on provider checkpoint 2024-03-07 07:48:23 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 2024-03-07 07:48:23 proc_lsst.multi:201 [INFO] cancelling 16887791 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887791 on provider escience_a40 2024-03-07 07:48:23 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:48:23 proc_lsst.multi:201 [INFO] cancelling 21546 on provider local proc_lsst.multi INFO: cancelling 21546 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 21546 2024-03-07 07:48:23 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro 2024-03-07 07:48:23 proc_lsst.multi:201 [INFO] cancelling 16887751 on provider astro proc_lsst.multi INFO: cancelling 16887751 on provider astro 2024-03-07 07:48:23 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 2024-03-07 07:48:23 proc_lsst.multi:201 [INFO] cancelling 16887805 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887805 on provider escience_a40 2024-03-07 07:48:23 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro 2024-03-07 07:48:23 proc_lsst.multi:201 [INFO] cancelling 16887756 on provider astro proc_lsst.multi INFO: cancelling 16887756 on provider astro 2024-03-07 07:48:23 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 2024-03-07 07:48:23 proc_lsst.multi:201 [INFO] cancelling 16887797 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887797 on provider escience_a40 2024-03-07 07:48:23 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 2024-03-07 07:48:23 proc_lsst.multi:201 [INFO] cancelling 16887799 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887799 on provider escience_a40 2024-03-07 07:48:23 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro 2024-03-07 07:48:23 proc_lsst.multi:201 [INFO] cancelling 16887760 on provider astro proc_lsst.multi INFO: cancelling 16887760 on provider astro 2024-03-07 07:48:24 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 2024-03-07 07:48:24 proc_lsst.multi:201 [INFO] cancelling 16887786 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887786 on provider escience_a40 2024-03-07 07:48:24 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint 2024-03-07 07:48:24 proc_lsst.multi:201 [INFO] cancelling 16887810 on provider checkpoint proc_lsst.multi INFO: cancelling 16887810 on provider checkpoint 2024-03-07 07:48:24 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro 2024-03-07 07:48:24 proc_lsst.multi:201 [INFO] cancelling 16887762 on provider astro proc_lsst.multi INFO: cancelling 16887762 on provider astro 2024-03-07 07:48:24 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 2024-03-07 07:48:24 proc_lsst.multi:201 [INFO] cancelling 16887796 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887796 on provider escience_a40 2024-03-07 07:48:24 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 2024-03-07 07:48:24 proc_lsst.multi:201 [INFO] cancelling 16887790 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887790 on provider escience_a40 2024-03-07 07:48:24 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro 2024-03-07 07:48:24 proc_lsst.multi:201 [INFO] cancelling 16887754 on provider astro proc_lsst.multi INFO: cancelling 16887754 on provider astro 2024-03-07 07:48:24 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 2024-03-07 07:48:24 proc_lsst.multi:201 [INFO] cancelling 16887789 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887789 on provider escience_a40 2024-03-07 07:48:24 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro 2024-03-07 07:48:24 proc_lsst.multi:201 [INFO] cancelling 16887770 on provider astro proc_lsst.multi INFO: cancelling 16887770 on provider astro 2024-03-07 07:48:24 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro 2024-03-07 07:48:24 proc_lsst.multi:201 [INFO] cancelling 16887753 on provider astro proc_lsst.multi INFO: cancelling 16887753 on provider astro 2024-03-07 07:48:25 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro 2024-03-07 07:48:25 proc_lsst.multi:201 [INFO] cancelling 16887757 on provider astro proc_lsst.multi INFO: cancelling 16887757 on provider astro 2024-03-07 07:48:25 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro 2024-03-07 07:48:25 proc_lsst.multi:201 [INFO] cancelling 16887779 on provider astro proc_lsst.multi INFO: cancelling 16887779 on provider astro 2024-03-07 07:48:25 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro 2024-03-07 07:48:25 proc_lsst.multi:201 [INFO] cancelling 16887768 on provider astro proc_lsst.multi INFO: cancelling 16887768 on provider astro parsl.dataflow.dflow INFO: Shutting down executor multi 2024-03-07 07:48:25 proc_lsst.multi:40 [INFO] Cancelling all provider resources proc_lsst.multi INFO: Cancelling all provider resources 2024-03-07 07:48:25 proc_lsst.multi:47 [INFO] new jobs since last cancel ['16887806', '16887807', '16887808', '16887809', '16887810', '16887811', '16887812', '16887813', '16887814', '16887815', '16887816', '16887817', '16887818', '16887819', '16887820', '16887821', '16887822', '16887823', '16887824', '16887825', '16887826', '16887827', '16887749', '16887750', '16887751', '16887752', '16887753', '16887754', '16887755', '16887756', '16887757', '16887758', '16887759', '16887760', '16887761', '16887762', '16887763', '16887766', '16887767', '16887768', '16887769', '16887770', '16887771', '16887772', '16887773', '16887774', '16887775', '16887776', '16887777', '16887778', '16887779', '16887780', '16887786', '16887787', '16887788', '16887789', '16887790', '16887791', '16887792', '16887793', '16887794', '16887795', '16887796', '16887797', '16887798', '16887799', '16887800', '16887801', '16887802', '16887803', '16887804', '16887805', '16887781', '16887782', '16887783', '16887784', '16887785', '21546'] proc_lsst.multi INFO: new jobs since last cancel ['16887806', '16887807', '16887808', '16887809', '16887810', '16887811', '16887812', '16887813', '16887814', '16887815', '16887816', '16887817', '16887818', '16887819', '16887820', '16887821', '16887822', '16887823', '16887824', '16887825', '16887826', '16887827', '16887749', '16887750', '16887751', '16887752', '16887753', '16887754', '16887755', '16887756', '16887757', '16887758', '16887759', '16887760', '16887761', '16887762', '16887763', '16887766', '16887767', '16887768', '16887769', '16887770', '16887771', '16887772', '16887773', '16887774', '16887775', '16887776', '16887777', '16887778', '16887779', '16887780', '16887786', '16887787', '16887788', '16887789', '16887790', '16887791', '16887792', '16887793', '16887794', '16887795', '16887796', '16887797', '16887798', '16887799', '16887800', '16887801', '16887802', '16887803', '16887804', '16887805', '16887781', '16887782', '16887783', '16887784', '16887785', '21546'] 2024-03-07 07:48:25 proc_lsst.multi:146 [INFO] found job 16887806 in provider checkpoint proc_lsst.multi INFO: found job 16887806 in provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:201 [INFO] cancelling 16887806 on provider checkpoint proc_lsst.multi INFO: cancelling 16887806 on provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:146 [INFO] found job 16887807 in provider checkpoint proc_lsst.multi INFO: found job 16887807 in provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:201 [INFO] cancelling 16887807 on provider checkpoint proc_lsst.multi INFO: cancelling 16887807 on provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:146 [INFO] found job 16887808 in provider checkpoint proc_lsst.multi INFO: found job 16887808 in provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:201 [INFO] cancelling 16887808 on provider checkpoint proc_lsst.multi INFO: cancelling 16887808 on provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:146 [INFO] found job 16887809 in provider checkpoint proc_lsst.multi INFO: found job 16887809 in provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:201 [INFO] cancelling 16887809 on provider checkpoint proc_lsst.multi INFO: cancelling 16887809 on provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:146 [INFO] found job 16887810 in provider checkpoint proc_lsst.multi INFO: found job 16887810 in provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:201 [INFO] cancelling 16887810 on provider checkpoint proc_lsst.multi INFO: cancelling 16887810 on provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:146 [INFO] found job 16887811 in provider checkpoint proc_lsst.multi INFO: found job 16887811 in provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:201 [INFO] cancelling 16887811 on provider checkpoint proc_lsst.multi INFO: cancelling 16887811 on provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:146 [INFO] found job 16887812 in provider checkpoint proc_lsst.multi INFO: found job 16887812 in provider checkpoint 2024-03-07 07:48:25 proc_lsst.multi:201 [INFO] cancelling 16887812 on provider checkpoint proc_lsst.multi INFO: cancelling 16887812 on provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:146 [INFO] found job 16887813 in provider checkpoint proc_lsst.multi INFO: found job 16887813 in provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:201 [INFO] cancelling 16887813 on provider checkpoint proc_lsst.multi INFO: cancelling 16887813 on provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:146 [INFO] found job 16887814 in provider checkpoint proc_lsst.multi INFO: found job 16887814 in provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:201 [INFO] cancelling 16887814 on provider checkpoint proc_lsst.multi INFO: cancelling 16887814 on provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:146 [INFO] found job 16887815 in provider checkpoint proc_lsst.multi INFO: found job 16887815 in provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:201 [INFO] cancelling 16887815 on provider checkpoint proc_lsst.multi INFO: cancelling 16887815 on provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:146 [INFO] found job 16887816 in provider checkpoint proc_lsst.multi INFO: found job 16887816 in provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:201 [INFO] cancelling 16887816 on provider checkpoint proc_lsst.multi INFO: cancelling 16887816 on provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:146 [INFO] found job 16887817 in provider checkpoint proc_lsst.multi INFO: found job 16887817 in provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:201 [INFO] cancelling 16887817 on provider checkpoint proc_lsst.multi INFO: cancelling 16887817 on provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:146 [INFO] found job 16887818 in provider checkpoint proc_lsst.multi INFO: found job 16887818 in provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:201 [INFO] cancelling 16887818 on provider checkpoint proc_lsst.multi INFO: cancelling 16887818 on provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:146 [INFO] found job 16887819 in provider checkpoint proc_lsst.multi INFO: found job 16887819 in provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:201 [INFO] cancelling 16887819 on provider checkpoint proc_lsst.multi INFO: cancelling 16887819 on provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:146 [INFO] found job 16887820 in provider checkpoint proc_lsst.multi INFO: found job 16887820 in provider checkpoint 2024-03-07 07:48:26 proc_lsst.multi:201 [INFO] cancelling 16887820 on provider checkpoint proc_lsst.multi INFO: cancelling 16887820 on provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:146 [INFO] found job 16887821 in provider checkpoint proc_lsst.multi INFO: found job 16887821 in provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:201 [INFO] cancelling 16887821 on provider checkpoint proc_lsst.multi INFO: cancelling 16887821 on provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:146 [INFO] found job 16887822 in provider checkpoint proc_lsst.multi INFO: found job 16887822 in provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:201 [INFO] cancelling 16887822 on provider checkpoint proc_lsst.multi INFO: cancelling 16887822 on provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:146 [INFO] found job 16887823 in provider checkpoint proc_lsst.multi INFO: found job 16887823 in provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:201 [INFO] cancelling 16887823 on provider checkpoint proc_lsst.multi INFO: cancelling 16887823 on provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:146 [INFO] found job 16887824 in provider checkpoint proc_lsst.multi INFO: found job 16887824 in provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:201 [INFO] cancelling 16887824 on provider checkpoint proc_lsst.multi INFO: cancelling 16887824 on provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:146 [INFO] found job 16887825 in provider checkpoint proc_lsst.multi INFO: found job 16887825 in provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:201 [INFO] cancelling 16887825 on provider checkpoint proc_lsst.multi INFO: cancelling 16887825 on provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:146 [INFO] found job 16887826 in provider checkpoint proc_lsst.multi INFO: found job 16887826 in provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:201 [INFO] cancelling 16887826 on provider checkpoint proc_lsst.multi INFO: cancelling 16887826 on provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:146 [INFO] found job 16887827 in provider checkpoint proc_lsst.multi INFO: found job 16887827 in provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:201 [INFO] cancelling 16887827 on provider checkpoint proc_lsst.multi INFO: cancelling 16887827 on provider checkpoint 2024-03-07 07:48:27 proc_lsst.multi:146 [INFO] found job 16887749 in provider astro proc_lsst.multi INFO: found job 16887749 in provider astro 2024-03-07 07:48:27 proc_lsst.multi:201 [INFO] cancelling 16887749 on provider astro proc_lsst.multi INFO: cancelling 16887749 on provider astro 2024-03-07 07:48:27 proc_lsst.multi:146 [INFO] found job 16887750 in provider astro proc_lsst.multi INFO: found job 16887750 in provider astro 2024-03-07 07:48:27 proc_lsst.multi:201 [INFO] cancelling 16887750 on provider astro proc_lsst.multi INFO: cancelling 16887750 on provider astro 2024-03-07 07:48:27 proc_lsst.multi:146 [INFO] found job 16887751 in provider astro proc_lsst.multi INFO: found job 16887751 in provider astro 2024-03-07 07:48:27 proc_lsst.multi:201 [INFO] cancelling 16887751 on provider astro proc_lsst.multi INFO: cancelling 16887751 on provider astro 2024-03-07 07:48:27 proc_lsst.multi:146 [INFO] found job 16887752 in provider astro proc_lsst.multi INFO: found job 16887752 in provider astro 2024-03-07 07:48:27 proc_lsst.multi:201 [INFO] cancelling 16887752 on provider astro proc_lsst.multi INFO: cancelling 16887752 on provider astro 2024-03-07 07:48:28 proc_lsst.multi:146 [INFO] found job 16887753 in provider astro proc_lsst.multi INFO: found job 16887753 in provider astro 2024-03-07 07:48:28 proc_lsst.multi:201 [INFO] cancelling 16887753 on provider astro proc_lsst.multi INFO: cancelling 16887753 on provider astro 2024-03-07 07:48:28 proc_lsst.multi:146 [INFO] found job 16887754 in provider astro proc_lsst.multi INFO: found job 16887754 in provider astro 2024-03-07 07:48:28 proc_lsst.multi:201 [INFO] cancelling 16887754 on provider astro proc_lsst.multi INFO: cancelling 16887754 on provider astro 2024-03-07 07:48:28 proc_lsst.multi:146 [INFO] found job 16887755 in provider astro proc_lsst.multi INFO: found job 16887755 in provider astro 2024-03-07 07:48:28 proc_lsst.multi:201 [INFO] cancelling 16887755 on provider astro proc_lsst.multi INFO: cancelling 16887755 on provider astro 2024-03-07 07:48:28 proc_lsst.multi:146 [INFO] found job 16887756 in provider astro proc_lsst.multi INFO: found job 16887756 in provider astro 2024-03-07 07:48:28 proc_lsst.multi:201 [INFO] cancelling 16887756 on provider astro proc_lsst.multi INFO: cancelling 16887756 on provider astro 2024-03-07 07:48:28 proc_lsst.multi:146 [INFO] found job 16887757 in provider astro proc_lsst.multi INFO: found job 16887757 in provider astro 2024-03-07 07:48:28 proc_lsst.multi:201 [INFO] cancelling 16887757 on provider astro proc_lsst.multi INFO: cancelling 16887757 on provider astro 2024-03-07 07:48:28 proc_lsst.multi:146 [INFO] found job 16887758 in provider astro proc_lsst.multi INFO: found job 16887758 in provider astro 2024-03-07 07:48:28 proc_lsst.multi:201 [INFO] cancelling 16887758 on provider astro proc_lsst.multi INFO: cancelling 16887758 on provider astro 2024-03-07 07:48:28 proc_lsst.multi:146 [INFO] found job 16887759 in provider astro proc_lsst.multi INFO: found job 16887759 in provider astro 2024-03-07 07:48:28 proc_lsst.multi:201 [INFO] cancelling 16887759 on provider astro proc_lsst.multi INFO: cancelling 16887759 on provider astro 2024-03-07 07:48:28 proc_lsst.multi:146 [INFO] found job 16887760 in provider astro proc_lsst.multi INFO: found job 16887760 in provider astro 2024-03-07 07:48:28 proc_lsst.multi:201 [INFO] cancelling 16887760 on provider astro proc_lsst.multi INFO: cancelling 16887760 on provider astro 2024-03-07 07:48:28 proc_lsst.multi:146 [INFO] found job 16887761 in provider astro proc_lsst.multi INFO: found job 16887761 in provider astro 2024-03-07 07:48:28 proc_lsst.multi:201 [INFO] cancelling 16887761 on provider astro proc_lsst.multi INFO: cancelling 16887761 on provider astro 2024-03-07 07:48:29 proc_lsst.multi:146 [INFO] found job 16887762 in provider astro proc_lsst.multi INFO: found job 16887762 in provider astro 2024-03-07 07:48:29 proc_lsst.multi:201 [INFO] cancelling 16887762 on provider astro proc_lsst.multi INFO: cancelling 16887762 on provider astro 2024-03-07 07:48:29 proc_lsst.multi:146 [INFO] found job 16887763 in provider astro proc_lsst.multi INFO: found job 16887763 in provider astro 2024-03-07 07:48:29 proc_lsst.multi:201 [INFO] cancelling 16887763 on provider astro proc_lsst.multi INFO: cancelling 16887763 on provider astro 2024-03-07 07:48:29 proc_lsst.multi:146 [INFO] found job 16887766 in provider astro proc_lsst.multi INFO: found job 16887766 in provider astro 2024-03-07 07:48:29 proc_lsst.multi:201 [INFO] cancelling 16887766 on provider astro proc_lsst.multi INFO: cancelling 16887766 on provider astro 2024-03-07 07:48:29 proc_lsst.multi:146 [INFO] found job 16887767 in provider astro proc_lsst.multi INFO: found job 16887767 in provider astro 2024-03-07 07:48:29 proc_lsst.multi:201 [INFO] cancelling 16887767 on provider astro proc_lsst.multi INFO: cancelling 16887767 on provider astro 2024-03-07 07:48:29 proc_lsst.multi:146 [INFO] found job 16887768 in provider astro proc_lsst.multi INFO: found job 16887768 in provider astro 2024-03-07 07:48:29 proc_lsst.multi:201 [INFO] cancelling 16887768 on provider astro proc_lsst.multi INFO: cancelling 16887768 on provider astro 2024-03-07 07:48:29 proc_lsst.multi:146 [INFO] found job 16887769 in provider astro proc_lsst.multi INFO: found job 16887769 in provider astro 2024-03-07 07:48:29 proc_lsst.multi:201 [INFO] cancelling 16887769 on provider astro proc_lsst.multi INFO: cancelling 16887769 on provider astro 2024-03-07 07:48:29 proc_lsst.multi:146 [INFO] found job 16887770 in provider astro proc_lsst.multi INFO: found job 16887770 in provider astro 2024-03-07 07:48:29 proc_lsst.multi:201 [INFO] cancelling 16887770 on provider astro proc_lsst.multi INFO: cancelling 16887770 on provider astro 2024-03-07 07:48:29 proc_lsst.multi:146 [INFO] found job 16887771 in provider astro proc_lsst.multi INFO: found job 16887771 in provider astro 2024-03-07 07:48:29 proc_lsst.multi:201 [INFO] cancelling 16887771 on provider astro proc_lsst.multi INFO: cancelling 16887771 on provider astro 2024-03-07 07:48:29 proc_lsst.multi:146 [INFO] found job 16887772 in provider astro proc_lsst.multi INFO: found job 16887772 in provider astro 2024-03-07 07:48:29 proc_lsst.multi:201 [INFO] cancelling 16887772 on provider astro proc_lsst.multi INFO: cancelling 16887772 on provider astro 2024-03-07 07:48:29 proc_lsst.multi:146 [INFO] found job 16887773 in provider astro proc_lsst.multi INFO: found job 16887773 in provider astro 2024-03-07 07:48:29 proc_lsst.multi:201 [INFO] cancelling 16887773 on provider astro proc_lsst.multi INFO: cancelling 16887773 on provider astro 2024-03-07 07:48:29 proc_lsst.multi:146 [INFO] found job 16887774 in provider astro proc_lsst.multi INFO: found job 16887774 in provider astro 2024-03-07 07:48:29 proc_lsst.multi:201 [INFO] cancelling 16887774 on provider astro proc_lsst.multi INFO: cancelling 16887774 on provider astro 2024-03-07 07:48:30 proc_lsst.multi:146 [INFO] found job 16887775 in provider astro proc_lsst.multi INFO: found job 16887775 in provider astro 2024-03-07 07:48:30 proc_lsst.multi:201 [INFO] cancelling 16887775 on provider astro proc_lsst.multi INFO: cancelling 16887775 on provider astro 2024-03-07 07:48:30 proc_lsst.multi:146 [INFO] found job 16887776 in provider astro proc_lsst.multi INFO: found job 16887776 in provider astro 2024-03-07 07:48:30 proc_lsst.multi:201 [INFO] cancelling 16887776 on provider astro proc_lsst.multi INFO: cancelling 16887776 on provider astro 2024-03-07 07:48:30 proc_lsst.multi:146 [INFO] found job 16887777 in provider astro proc_lsst.multi INFO: found job 16887777 in provider astro 2024-03-07 07:48:30 proc_lsst.multi:201 [INFO] cancelling 16887777 on provider astro proc_lsst.multi INFO: cancelling 16887777 on provider astro 2024-03-07 07:48:30 proc_lsst.multi:146 [INFO] found job 16887778 in provider astro proc_lsst.multi INFO: found job 16887778 in provider astro 2024-03-07 07:48:30 proc_lsst.multi:201 [INFO] cancelling 16887778 on provider astro proc_lsst.multi INFO: cancelling 16887778 on provider astro 2024-03-07 07:48:30 proc_lsst.multi:146 [INFO] found job 16887779 in provider astro proc_lsst.multi INFO: found job 16887779 in provider astro 2024-03-07 07:48:30 proc_lsst.multi:201 [INFO] cancelling 16887779 on provider astro proc_lsst.multi INFO: cancelling 16887779 on provider astro 2024-03-07 07:48:30 proc_lsst.multi:146 [INFO] found job 16887780 in provider astro proc_lsst.multi INFO: found job 16887780 in provider astro 2024-03-07 07:48:30 proc_lsst.multi:201 [INFO] cancelling 16887780 on provider astro proc_lsst.multi INFO: cancelling 16887780 on provider astro 2024-03-07 07:48:32 proc_lsst.multi:146 [INFO] found job 16887786 in provider escience_a40 proc_lsst.multi INFO: found job 16887786 in provider escience_a40 2024-03-07 07:48:32 proc_lsst.multi:201 [INFO] cancelling 16887786 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887786 on provider escience_a40 2024-03-07 07:48:32 proc_lsst.multi:146 [INFO] found job 16887787 in provider escience_a40 proc_lsst.multi INFO: found job 16887787 in provider escience_a40 2024-03-07 07:48:32 proc_lsst.multi:201 [INFO] cancelling 16887787 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887787 on provider escience_a40 2024-03-07 07:48:32 proc_lsst.multi:146 [INFO] found job 16887788 in provider escience_a40 proc_lsst.multi INFO: found job 16887788 in provider escience_a40 2024-03-07 07:48:32 proc_lsst.multi:201 [INFO] cancelling 16887788 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887788 on provider escience_a40 2024-03-07 07:48:32 proc_lsst.multi:146 [INFO] found job 16887789 in provider escience_a40 proc_lsst.multi INFO: found job 16887789 in provider escience_a40 2024-03-07 07:48:32 proc_lsst.multi:201 [INFO] cancelling 16887789 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887789 on provider escience_a40 2024-03-07 07:48:32 proc_lsst.multi:146 [INFO] found job 16887790 in provider escience_a40 proc_lsst.multi INFO: found job 16887790 in provider escience_a40 2024-03-07 07:48:32 proc_lsst.multi:201 [INFO] cancelling 16887790 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887790 on provider escience_a40 2024-03-07 07:48:32 proc_lsst.multi:146 [INFO] found job 16887791 in provider escience_a40 proc_lsst.multi INFO: found job 16887791 in provider escience_a40 2024-03-07 07:48:32 proc_lsst.multi:201 [INFO] cancelling 16887791 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887791 on provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:146 [INFO] found job 16887792 in provider escience_a40 proc_lsst.multi INFO: found job 16887792 in provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:201 [INFO] cancelling 16887792 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887792 on provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:146 [INFO] found job 16887793 in provider escience_a40 proc_lsst.multi INFO: found job 16887793 in provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:201 [INFO] cancelling 16887793 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887793 on provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:146 [INFO] found job 16887794 in provider escience_a40 proc_lsst.multi INFO: found job 16887794 in provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:201 [INFO] cancelling 16887794 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887794 on provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:146 [INFO] found job 16887795 in provider escience_a40 proc_lsst.multi INFO: found job 16887795 in provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:201 [INFO] cancelling 16887795 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887795 on provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:146 [INFO] found job 16887796 in provider escience_a40 proc_lsst.multi INFO: found job 16887796 in provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:201 [INFO] cancelling 16887796 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887796 on provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:146 [INFO] found job 16887797 in provider escience_a40 proc_lsst.multi INFO: found job 16887797 in provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:201 [INFO] cancelling 16887797 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887797 on provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:146 [INFO] found job 16887798 in provider escience_a40 proc_lsst.multi INFO: found job 16887798 in provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:201 [INFO] cancelling 16887798 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887798 on provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:146 [INFO] found job 16887799 in provider escience_a40 proc_lsst.multi INFO: found job 16887799 in provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:201 [INFO] cancelling 16887799 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887799 on provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:146 [INFO] found job 16887800 in provider escience_a40 proc_lsst.multi INFO: found job 16887800 in provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:201 [INFO] cancelling 16887800 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887800 on provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:146 [INFO] found job 16887801 in provider escience_a40 proc_lsst.multi INFO: found job 16887801 in provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:201 [INFO] cancelling 16887801 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887801 on provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:146 [INFO] found job 16887802 in provider escience_a40 proc_lsst.multi INFO: found job 16887802 in provider escience_a40 2024-03-07 07:48:33 proc_lsst.multi:201 [INFO] cancelling 16887802 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887802 on provider escience_a40 2024-03-07 07:48:34 proc_lsst.multi:146 [INFO] found job 16887803 in provider escience_a40 proc_lsst.multi INFO: found job 16887803 in provider escience_a40 2024-03-07 07:48:34 proc_lsst.multi:201 [INFO] cancelling 16887803 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887803 on provider escience_a40 2024-03-07 07:48:34 proc_lsst.multi:146 [INFO] found job 16887804 in provider escience_a40 proc_lsst.multi INFO: found job 16887804 in provider escience_a40 2024-03-07 07:48:34 proc_lsst.multi:201 [INFO] cancelling 16887804 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887804 on provider escience_a40 2024-03-07 07:48:34 proc_lsst.multi:146 [INFO] found job 16887805 in provider escience_a40 proc_lsst.multi INFO: found job 16887805 in provider escience_a40 2024-03-07 07:48:34 proc_lsst.multi:201 [INFO] cancelling 16887805 on provider escience_a40 proc_lsst.multi INFO: cancelling 16887805 on provider escience_a40 2024-03-07 07:48:34 proc_lsst.multi:146 [INFO] found job 16887781 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887781 in provider escience_rtx6k 2024-03-07 07:48:34 proc_lsst.multi:201 [INFO] cancelling 16887781 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16887781 on provider escience_rtx6k 2024-03-07 07:48:34 proc_lsst.multi:146 [INFO] found job 16887782 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887782 in provider escience_rtx6k 2024-03-07 07:48:34 proc_lsst.multi:201 [INFO] cancelling 16887782 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16887782 on provider escience_rtx6k 2024-03-07 07:48:34 proc_lsst.multi:146 [INFO] found job 16887783 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887783 in provider escience_rtx6k 2024-03-07 07:48:34 proc_lsst.multi:201 [INFO] cancelling 16887783 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16887783 on provider escience_rtx6k 2024-03-07 07:48:34 proc_lsst.multi:146 [INFO] found job 16887784 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887784 in provider escience_rtx6k 2024-03-07 07:48:34 proc_lsst.multi:201 [INFO] cancelling 16887784 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16887784 on provider escience_rtx6k 2024-03-07 07:48:34 proc_lsst.multi:146 [INFO] found job 16887785 in provider escience_rtx6k proc_lsst.multi INFO: found job 16887785 in provider escience_rtx6k 2024-03-07 07:48:34 proc_lsst.multi:201 [INFO] cancelling 16887785 on provider escience_rtx6k proc_lsst.multi INFO: cancelling 16887785 on provider escience_rtx6k 2024-03-07 07:48:34 proc_lsst.multi:146 [INFO] found job 21546 in provider local proc_lsst.multi INFO: found job 21546 in provider local 2024-03-07 07:48:34 proc_lsst.multi:201 [INFO] cancelling 21546 on provider local proc_lsst.multi INFO: cancelling 21546 on provider local parsl.providers.local.local DEBUG: Terminating job/proc_id: 21546 parsl.providers.local.local WARNING: Failed to kill PID: 21546 and child processes on local 2024-03-07 07:48:35 proc_lsst.multi:50 [INFO] no new jobs since last cancel, resuming executor shutdown proc_lsst.multi INFO: no new jobs since last cancel, resuming executor shutdown parsl.executors.high_throughput.executor INFO: Attempting HighThroughputExecutor shutdown parsl.executors.high_throughput.executor INFO: Finished HighThroughputExecutor shutdown attempt parsl.dataflow.dflow INFO: Shut down executor multi parsl.dataflow.dflow INFO: Shutting down executor _parsl_internal parsl.executors.threads DEBUG: Shutting down executor, which involves waiting for running tasks to complete parsl.executors.threads DEBUG: Done with executor shutdown parsl.dataflow.dflow INFO: Shut down executor _parsl_internal parsl.dataflow.dflow INFO: Terminated executors parsl.dataflow.dflow INFO: DFK cleanup complete parsl.process_loggers DEBUG: Normal ending for cleanup on thread MainThread lsst.ctrl.bps.submit INFO: Completed submitting to a workflow management system: Took 1400.4192 seconds lsst.ctrl.bps.drivers INFO: Run 'DEEP_20220823_B1d_science#step1_20240307T072452Z' submitted for execution with id 'None' lsst.ctrl.bps.drivers INFO: Completed submit stage: Took 1400.4250 seconds; current memory usage: 0.386 Gibyte, delta: 0.008 Gibyte, peak delta: 0.008 Gibyte lsst.ctrl.bps.drivers INFO: Completed entire submission process: Took 1419.8228 seconds; current memory usage: 0.386 Gibyte, delta: 0.248 Gibyte, peak delta: 0.248 Gibyte lsst.ctrl.bps.drivers INFO: Peak memory usage for bps process 0.386 Gibyte (main), 0.386 Gibyte (largest child process) Run Id: None Run Name: DEEP_20220823_B1d_science#step1_20240307T072452Z parsl.dataflow.dflow INFO: python process is exiting, but DFK has already been cleaned up